Previously in this series - ,

OpenAI’s pretrained Gpt2 models have been all the rage in nlp model finetunning. Sub Simulators, AI Dungeons, Poetry, Chess, Cards Against Humanity - you name it.

My particular interest has been in applying it to my personal chat data, so I can talk to arbitrary friends and more importantly myself whenever I want to. Why? Well, sometimes if you don’t hear voices you have to go and program the voices into being. More seriously, I wished to be able to talk with a version of myself so I can get an outside view of how I think, for analysing my tendencies and problems, rubber ducking, self-therapy and - let’s face it - most importantly for fun.

I won’t go into detail describing the Colab with Facebook data, as it is almost the same as the one used with the 117M version which I describe here and here. We are again using nsheppered’s GPT training code as a base with shawwwn’s fork allowing us to train the whole of gpt-1.5b on a TPU - which means we can do so for free on Colab again! I’ve only added a little bit more so we can talk interactively with gpt2, have it respond to us as a specific person and go through the whole general back and forth which we typically call ‘having a conversation’.

You can train your own in the new GPT2-1.5b collab here. Make sure to click Runtime> Change Runtime type> TPU

Note: It is a lot easier to train if you pay a few bucks for so you can fit it all easily on Drive. A checkpoint of the model comes to ~6 gb, so if you only keep one checkpoint you can do it without paying - it is just more hassle.

Note 2: It took me over a month of training to get it here using just Colab

Self-Chat Samples

Motivation

The idea of modelling a person fascinates me, and as language models have been becoming better and better it genuinely seems like they capture more and more of a person’s style and tendencies. Honestly, I don’t even need more motivation than that to want to do it - the only real question is who to model as better and bigger models come out?

Now, I have to admit - my first choice would be someone like Von Neumann but that’s not someone on whom I have the right type of conversational data to do well (although there are ways to tackle the problem). The person who I am guaranteed to have data on, AND who is still very useful and interesting subject of study for myself is, you guessed it, myself.

Is it useful though? Don’t I already have access to a pretty good, some might say perfect simulation of myself from the get-go? I do. However, if I can have a second one of me to bounce ideas with I’d take it. If I can analyse my problems from the outside rather than from the inside, I’d take that, too.

After playing with the results, and extrapolating to what the next generation of models will be able to do (gpt-2 is already not the newest and best architecture), I think a lot of people might at some point want a version of themselves to talk to, bounce ideas with, get help from and so on.

A Few More Words

I’ve spent some time talking to the irc chat bots of nshepperd and shawwn (both trained on the same data) and had a lot of fun with it. Talking to myself though has been highly addictive, and I’ve been running a Colab where I occasionally shoot myself a question or go into a deep conversation with my gpt2 clone for days now. It’s …interesting.

One thing I didn’t realize though was how personal it is - it’s all trained on my facebook messenger data, where I often talk with my closest friends. It takes very little for gpt to start talking about intimate topics which I’d neither want to share nor would they be as meaningful to anyone else.

Having said that the next are my last 7 unfiltered (I’ve only changed the names) conversations with it, where I am actively trying to avoid sensitive topics (I still had to restart it twice due to sensitive subjects).

Results

The end loss I reached (for whatever that is worth) was just over 0.6, substantially lower than with the smaller models. My dataset is still pretty small - 14mb, and the next step would be to add my personal data from other sources. Yet, it does quite well, though perhaps it wouldn’t seem as exciting to others.

Facebook

The way it works is - I write a sentence under the name of a friend from the dataset but here labeled as ‘Me’, pass it to the model while prompting it to start typing using my name - ‘Svilen Todorov’, (here labeled as ‘Gpt Me’). When it starts outputting something not by ‘Svilen Todorov’ I cut it off, and write myself. Then I feed that + the history so far to the model, which allows us to have an arbitrarily long coherent conversation.

Chat Samples:

Example 1

Me: How's work?

GPT Me: ive been working a bit but still not in the office

Me: I've been working, too. Can't solve this damn bug I have.

GPT Me: ive spent hours on it

GPT Me: but to no avail

GPT Me: im spending all my time on it

Me: I checked everything, yet it runs out of memory despite obvious leaks. hoooow

GPT Me: ive been wasting so much of my time on it tho

GPT Me: 16gb memory, 16gb video ram, 60 hours a week for 3 years

GPT Me: barely have anything open other than youtube and facebook

Me: oh it's a browser issue? man the web sure sounds bloated nowadays

GPT Me: ive tried everything

Me: did you try turning it on and off?

GPT Me: haha

GPT Me: ive changed it between view and off in the dropbox since the beginning, should be the same now as in your case and I doesnt seem to be working

GPT Me: and tried a few things

GPT Me: nothing seems to be working

GPT Me: i uploaded a new one though when you did

GPT Me: but pointless to try it now

Me: don't give up

GPT Me: ive uploaded a new one

GPT Me: doesnt run properly

GPT Me: etc.

GPT Me: still nothing

GPT Me: maybe in a few days when i have a few gigs under my belt it will run a bit better

GPT Me: and then in the mean time i can send you what i have

GPT Me: to use as you like

GPT Me: until then use whatever is in my dropbox

GPT Me: if you want i can even send you my complete open source build, so you can build it and send me what i use

Me: ok, let's do that

GPT Me: ive only sent you a part of the source so i can get the full thing

GPT Me: https://github.com/p5pit/pciusb

Me: I can't access it. Can you resend the link?

GPT Me: ive just changed the link in the source

Me: what's the new url?

GPT Me: ive just changed the name aboven you send it

GPT Me: https://github.com/p5pit/pciusb/tree/master/libs

GPT Me: but you need to update the links in the 'message' part

GPT Me: anyway whats up with you

Me: Not much, just reading

GPT Me: ive been at lectures 4-7

GPT Me: im helping a mate with some maths problems

GPT Me: and trying to read a paper since 5

Me: ok cool

GPT Me: ive not sure what it was on

GPT Me: Lyndon had problems with reading just finding the right words in random papers to use them to make a point

GPT Me: and it was on whether certain types of choice can build a successful models to fit the data to believe





Example 2

Me: hey how are you

GPT Me: ive quit and you know where to find me

Me: what did you quit

GPT Me: my job, ive given a few interviews here and there

GPT Me: and my boss is giving me shit for not doing more work during the day

GPT Me: but ive quit

GPT Me: he's so annoyed at me

GPT Me: thats a bit of an overreaction

Me: yeah you shouldnt have to work

GPT Me: ive been working since 8 am today

Me: oh, you quit your job and are working on getting a new one?

GPT Me: ive been trying to for a while now

GPT Me: and the previous one ends in oct

Me: what job are you looking for?

GPT Me: ive been searching for 'concierge' level stuff

GPT Me: i like working on cutting edge stuff

Me: so something in AI again?

GPT Me: ive been looking at it for 8 months at least

GPT Me: and they want you to start at 17 / 20 euro a day

GPT Me: with no holiday

GPT Me: which is pretty ridiculous

Me: that does seem like a ridiculously low salary

GPT Me: ive talked to people at home and abroad about it

GPT Me: and they were all 'wtf'

GPT Me: i havent even called in to my third favourite coffee place to try to get them to put me as a regular

Me: interesting

GPT Me: ive gone to application after meeting with them

GPT Me: :D

GPT Me: coffee and biscuits*

Me: how did it go

GPT Me: ive only went for a small place so far, startup

GPT Me: and they wanted me to make a profile there

GPT Me: and to tell them about the personal stuff like my salary and social security

GPT Me: and if i want it there it has to be on the phone

GPT Me: and so on





Example 3

Me: whats up

GPT Me: ive been having a lot of success with my programming but dunno if Ive succeed

GPT Me: i have a lot of stuff

GPT Me: not sure how many hours Im going to spend a day on it

Me: you are not working too much and getting burnt out are you?

GPT Me: ive had years with zero effort and amazing results

GPT Me: - especially when compared to other programmers

GPT Me: so yeah, probably a bit

Me: well, take it easy man

GPT Me: ive been doing it for a few hours a day for a few years now

GPT Me: i really want to move to machine learning though

GPT Me: and Ive started on that over a year ago

Me: what ML stuff do you want to do

GPT Me: ive done a little in R, but want to move to using things like neural nets for a lot more

GPT Me: ideally something which is more general

GPT Me: and less focused on just driving

Me: but what do you want to work on specifically

GPT Me: ive helped people a lot with getting them to focus on a specific problem

GPT Me: at this point

GPT Me: :D

Me: so what problem are YOU focusing on

GPT Me: ive helped a lot with getting them to focus on adding simple analytics

GPT Me: to their sites

GPT Me: and getting them to focus on just a few things

GPT Me: rather than whole things

GPT Me: but im moving more towards the latter

Me: hm

GPT Me: ive done a lot of the heavy lifting for them, by getting them to focus on a task that is either too hard or overly boring for them

GPT Me: but thats a bit of an unfair comparison

GPT Me: they have their own stuff

Me: and what do you have

GPT Me: ive done a lot, but they have their stuff

GPT Me: i have to move away from that

GPT Me: in a few months

GPT Me: and program something better

Me: like what

GPT Me: something general

GPT Me: agi





Example 4

GPT Me: ive done fuck all for an hour now

GPT Me: nothing

GPT Me: i shouldnt be working today

GPT Me: i was supposed to be working yesterday too but cba

GPT Me: just tired

Me: you seem to be tired all the time, man. everything ok?

GPT Me: ive been sleeping since 9

GPT Me: no, not really

GPT Me: my back hurts

GPT Me: and i hurt my back peacing out

GPT Me: but other than that im ok

GPT Me: i can do little but write

GPT Me: but i cant read and i havent done anything in days

GPT Me: i am so bored

GPT Me: my head is so bad

GPT Me: :/

Me: it happens - maybe take a break until you feel better

GPT Me: ive never been so annoyed by my head

GPT Me: i can barely use the computer

Me: damn that sucks

GPT Me: ive been worse

GPT Me: but i dont think its as bad as my back

GPT Me: which cant even move

Me: do you need any help?

GPT Me: ive been asking people to do shit for me

GPT Me: for a long time now

GPT Me: no help

GPT Me: people give me shit for not doing more

GPT Me: havent had a break until now

GPT Me: i feel so bad for not being able to pass out

Me: what have you tried for your sleeping problems so far?

GPT Me: ive been trying for a long time an antihistamines

GPT Me:beta-blockers

GPT Me:xanax

GPT Me:histamine

GPT Me:hainamines

GPT Me:hordamances

GPT Me:hordana

GPT Me:hordano

GPT Me:shrooms

GPT Me:whatever

GPT Me: not even going to read now

GPT Me: too tired

GPT Me: anything else

GPT Me: slowing myself down

GPT Me: exhausted

GPT Me: read the whole thing now

Me: I did but I am confused. And you shouldn't take shrooms to sleep.

GPT Me: i didnt really

GPT Me: its a whole thing





Example 5

Me: want to listen to something cool. send me a track

GPT Me: ive been putting tracks from acts I like on and off

GPT Me: into my library

Me: got any links to share?

GPT Me: ive been pushing some friends to listen to more stuff

GPT Me: so you can do that too now :D

GPT Me: https://www.youtube.com/watch?v=hfTPiS7Nmw

GPT Me: https://www.youtube.com/watch?v=QsXgEzx_GGk

GPT Me: and this other one is one of the earlier songs I liked from them

GPT Me: https://www.youtube.com/watch?v=Iqz02saPcoM

Me: cool! I like the cramps. Got any techno tho?

GPT Me: ive been trying to get people to listen to some of the stuff

Me: which stuff? what artists or labels?

GPT Me: ive been pushing mostly labelless stuff which are usually quite good

GPT Me: stuff by bez batteries and their cousin or whatever

GPT Me: and stuff by dvs1 and his cousin

GPT Me: or vatican shadow or whatever

GPT Me: ive sent you the first two

Me: cool





Example 6

Me: hey man, what are you up to this weekend?

GPT Me: ive tickets for a party on friday

GPT Me: wasnt expecting to go out this early

GPT Me: but got tickets during the sale

GPT Me: so im waking up early i guess

Me: cool, what's the party?

GPT Me: ive told people and they havent been able to guess

GPT Me: https://www.facebook.com/events/553455714529578/

GPT Me: its a bit weird

GPT Me: i have friends in charge

GPT Me: who i trust more than me

GPT Me: to tell me if i have a ticket

Me: looks good. who's playing?

GPT Me: ive asked them

GPT Me: they said they dont know

GPT Me: i hope i do

GPT Me: but they were saying it in a way that it sounds bad

GPT Me: :D

GPT Me: aufnahme'samenere

GPT Me: its a bit weird

GPT Me: the only guy that ive met like him is berghainarian

GPT Me: no wait that was mike

GPT Me: still ja

GPT Me: and its this group that does weird stuff

GPT Me: that guy does the sound, and looks kind of weird

Me: sounds pretty interesting really

GPT Me: ive read a little about him on wiki but thats it

GPT Me: he is apparently a maker

GPT Me: but doesnt really know how the word works

Me: do I know him?

GPT Me: ive met him and his brother at makerfaire

GPT Me: very briefly

GPT Me: but they were really high

GPT Me: and i dont really remember them

GPT Me: thats 2 months ago

GPT Me: :D

Me: fair enough





Example 7

Me: how would you describe yourself

GPT Me: ive been working in the same job for 3 and a half years now

GPT Me: relatively new to berlin

GPT Me: 25

GPT Me: 6 years as a programmer/machine learning person

GPT Me: pretty much all my time for the last 5 years has been spent on just learning machine learning and not programming really

GPT Me: and Im pretty sure that even if I was to move elsewhere and work with people from different fields, I would still not be able to keep up

GPT Me: I would spend most of my time on just learning the ropes and whatever else they often have free or cheap classes for everyone

GPT Me: that seems to be the general approach in many places

GPT Me: but I havent seen that approach work for me

GPT Me: i spend literally every day with people and projects that dont allow me to do more than that

Me: and what about your personal life and hobbies?

GPT Me: ive just been doing the same stuff ever since I moved to berlin a year ago

GPT Me: didnt really do much else than that and the online courses to where I am now

GPT Me: I used to spend my time more on random stuff

GPT Me: like programming, music, reading things like that

Me: what about going out

GPT Me: and its rained a lot the last few days which doesnt help either

GPT Me: https://www.facebook.com/photo.php?fbid=324979844210832&set=a.324977827544367.75530.14546_1858271655695&type=3&theater

Me: sounds fun





Conclusion

The conversations aren’t ideal but it’s a lot fun playing with it. Sadly the stuff I talk about on facebook is either too personal or generic ( e.g. arranging to meet people) so I am unsure how well I’ve managed to convey the fun of talking to it. Perhaps another sample of gpt2 talking as me, but with the irc bot mentioned earlier (and ran by nshepperd), might convey it. You can find some more of those samples on Shawn’s twitter + a lot more gpt fun from his collaborations with gwern.

I genuinely think that even with gpt2, let alone bigger and better models you can get a lot of use out of having a model of yourself you can talk to - and not just yourself. Playing with this has really made me realise the value of a service that can create a model of dead or dying relatives, and I want to eventually set up something for my loved ones - at minimum the data collection part.

It will be unfair to say that it is always great though - sometimes it talks about whatever it wants to and the conversation can be harder to steer, otherwise it follows my lead easily. From what I can see in my actual messages I do this a lot, so maybe it is just copying me. Additionally it sometimes talks about things that aren’t really True - like the back pain in Example 1, and if you play with the different parameters (top_k/top_p and temperature mainly) you can force it to go on long tirades which eventually become nonsensical. It’s definitely not perfect.. but it sure is fascinating to me.

Fun fact - there is a sentence in this post written entirely by the GPT version of Me. I wonder how easy it is to spot.