Probably a stupid question, but can't you split the import into batches instead of trying to do it in one shot?
It's already being done in batches. That's how the convert script operates to prevent itself from overloading the site and/or database.
The problem is not so much with the batching, the problem is that the phpBB setup contains a lot of 'hidden corruption'. In technical terms, this means table contents that don't match the table type, table contents that cannot be converted to the target table type, etc.