followed an example from https://github.com/huggingface/transformers/blob/master/examples/run_squad.py to fine tune a bert-base model to do question-answering, after training on SQuAD 1.1 data using free GPU on Colaboratory for 2 hours, final F1 score: 88.0
example input:
doc = "Once upon a time there was an old mother pig who had three little pigs and not enough food to feed them. So when they were old enough, she sent them out into the world to seek their fortunes. The first little pig was very lazy. He didn't want to work at all and he built his house out of straw. The second little pig worked a little bit harder but he was somewhat lazy too and he built his house out of sticks. Then, they sang and danced and played together the rest of the day. The third little pig worked hard all day and built his house with bricks. It was a sturdy house complete with a fine fireplace and chimney. It looked like it could withstand the strongest winds. The next day, a wolf happened to pass by the lane where the three little pigs lived; and he saw the straw house, and he smelled the pig inside. He thought the pig would make a mighty fine meal and his mouth began to water."
q = 'What did the pigs do for the rest of the day?'
example output:
'sang and danced and played together'