I wrote a guide previously called the big howto on everything, which outlined how to setup the system I wrote for SNS translations, livestreams, etc., but unfortunately it was very hard to follow and setup, so here's my second shot at it.

If you have any issues with the guide, feel free to message me, and I'll try to help :)

Introduction & Disclaimer

This is a relatively complex system. It allows you to track people's posts, stories, livestreams, etc. from a number of different social media websites, as well as automatically processing the data in any way you wish, including downloading posts/recording livestreams, or creating a bot that will reupload them to another social media website (such as LiveBot on discord).

With that being said… it was written organically. That is to say, I had absolutely no idea what I was doing when I wrote this. If you ever wonder why something is written the way it was, instead of [insert better way here], chances are it's because I was/am an idiot.

There are certain problems I have been able to fix (or plan on fixing soon), but there are a few others where fixing them would require rewriting a large portion of the system.

If you plan on setting this system up on your computer, here are a few (soft) requirements:

Linux or OSX. This system was written for and tested on Linux, but a few people got it running on OSX without much trouble It does run under Cygwin (allows you to run Linux programs on Windows), but it's a huge pain to setup, not to mention that high I/O performance is critical for downloading livestreams. Until Windows significantly improves its I/O performance (which might not happen anytime soon), I'd personally avoid using it.

Speaking of I/O, I'd recommend using an HDD. The system writes >50GB per day on my computer, which will wear out an SSD. If using Linux, you might want to play around with I/O schedulers if using mq. Personally I found bfq to work well enough for my needs.

Some knowledge of python/node.js, or at least some experience in tinkering with a unix-like system. While the system is relatively powerful, it's not very user-friendly, and I haven't written much documentation (other than these guides)



And a few notes:

If possible, consider using a spare computer for this. Running a server on your production computer prevents you from really being able to use your computer freely. With that being said, I'd recommend against running this on a server farm, Instagram apparently sees that as a warning sign.

Use a spare Instagram account, and be careful not to follow or (especially) unfollow too many people at once. Instagram has been known to flag accounts that do this.

So here is where the big disclaimer comes in:

If all you want is to record instagram lives, this is probably not the right solution for you!

While this does allow you to record instagram lives, it's a relatively complicated system, and will realistically be more of a weekend project (or longer) to setup. It can be setup in a couple of hours, but you would really have to know what you're doing.

If the other features this script provides doesn't sound worth the time investment, someone I know had success using notcammy's PyInstaLive, which supports recording multiple accounts using the -df option.

There are also other options, such as taengstagram's instagram-livestream-downloader, but from a quick look at the source code, it seems to only support recording a single account at once.

In any case, it's always a good idea to try multiple programs before settling on one :)

Overview

Now that's out of the way, here's a bird's eye view of the system:

RSSit: The main driver behind the entire system Basically, it uses either private or public APIs to on various social networks, and creates RSS feeds from them. Some examples: Instagram user posts (supports normal posts, stories, lives/replays IGTV, profile pictures) Instagram reels tray (the thing containing all the lives and stories at the top of Instagram, this is how lives are found and recorded) Periscope lives (rather difficult to setup though unfortunately) Twitter user posts/retweets Youtube lives and videos A bunch of others that may or may not still work Written in Python -- Wasn't a good idea. Very slow, memory issues come and go with python updates, overall was just incredibly painful to use Because of some memory leak that I've been trying to track down for almost a year, but still haven't found, you need to restart it every few days or else it will slowly fill up your memory.

webrssview: It's a feed viewer that you can run in your web browser. This is what runs RSSit. Any feed viewer would work, but this one does have a bunch of functionality that makes it quite useful for what I do. It's also needed for most of the khelpers scripts (see below) Pretty fast (mainly thanks to Mongo and not using Python), but I made the incredibly stupid mistake of having storing the feeds list as a single object. Mine weighs ~3MB, if JSON.stringify() ed. Performance and CPU usage isn't optimal, to say the least Fixing this would require rewriting a good chunk of webrssview, as well as khelpers. And require setting up a separate development environment for the entire system, waiting for livestreams, a ton of debugging…

dlscripts: The first project I wrote, contains a ton of scripts that probably don't work anymore, but the important ones are download.py and iglivedl.py download.py allows you to download images and videos from social media posts found by RSSit iglivedl.py will download Instagram livestreams. Technically it's just a DASH livestream downloader, but it's only been tested on Instagram. I wouldn't recommend to use it for anything else remlive.py allows you to remove old posts and livestreams automatically. download.py is hooked into RSSit via social_hooks (more details on that later), which will run iglivedl.py automatically for Instagram livestreams (unless no_livedl is set to true in RSSit). Written in Python, which caused problems with iglivedl.py in the past, as well as performance being suboptimal

khelpers: This one is completely optional, it's only really useful if you want to upload lives automatically to YouTube. Miscellaneous scripts for various tasks: Automatic youtube upload ( youtubeul.js ) Tools for removing content ID-blocked portions of lives ( remcopyright.js ) and rotating videos ( rotatevid.js ) Helpers for automating the non-translation parts of the SNS translations I used to(?) do ( sns.js ) The LiveBot discord bot ( discordbot.js for the server/bot, discordbot_send.js for the RSSit hook) Due to webrssview's feeds being in a single object, and discordbot_send.js being run every few seconds, CPU usage is needlessly high This could be (kind of) worked around by having some kind of server that would reload the feeds at a specified interval, and handle more specific/smaller queries sent by discordbot_send.js



How then does everything work together? With chains and duct tape. Or: Some connections are a little hackier than others.

Basically:

You use rssit's main page (localhost:8123) to generate feed URLs, and you put those in webrssview (under a specific directory structure, if you want to use khelpers)

Webrssview reloads RSSit feeds automatically

RSSit does the API queries and generates a feed, usually in the "social" format.

It then forks: The "social" format gets converted into the "feed" format, which in turn gets converted into rss/atom, and returned back to webrssview. Nothing further happens here. Each format can get hooked by an external program. In this case, we will hook dlscript's download.py (and optionally discordbot_send.js ) to the "social" format.

download.py will automatically download all the photos, videos, and livestreams the feed returned. If there are Instagram livestreams, it will call iglivedl.py

will automatically download all the photos, videos, and livestreams the feed returned. If there are Instagram livestreams, it will call Once iglivedl.py is completed, it will run the "live_hook" from tokens.json (this is the "duct tape" connection).

is completed, it will run the "live_hook" from (this is the "duct tape" connection). live_hook will be set to khelpers' youtubeul.js . It uses parse_feeds.js to parse webrssview's feeds, as well as a file called feeds.toml to find all of the groups and members in the database.

. It uses to parse webrssview's feeds, as well as a file called to find all of the groups and members in the database. After it has uploaded the video to youtube, it will send a desktop notification, which can optionally be handled by rssit.

If you don't understand everything, that's completely fine, it will (hopefully) make sense later. This is just to orient you so you understand how it works when you set it up (something I didn't do in the previous guide).

Speaking of which…

Installation

First, make sure you have the required dependencies installed and properly setup:

Node.js + NPM (webrssview, khelpers)

Python 3 + Pip (rssit, dlscripts)

MongoDB (webrssview, discord bot)

Redis (rssit, dlscripts) This is technically optional, but it significantly improves performance.

Youtube-DL (dlscripts)

FFmpeg (dlscripts, khelpers)

Next, either download or git clone each repository (I'd recommend cloning because it allows you to easily update them in the future):

rssit

webrssview

dlscripts

khelpers (skip this if you're not interested in automatically uploading to YouTube)

Now install all of their required dependencies:

cd webrssview npm install cd ../khelpers npm install cd ../rssit sudo pip3 install redis # technically optional, see note above pip3 install -r requirements.txt cd ../dlscripts pip3 install -r requirements.txt

Setup (1/3)

Each project has a unique way of configuring it. "Organic" development at its finest I guess.

RSSit

Create a file called ~/.config/rssit/config.ini , with the following contents:

[default] social_hooks = python3 /path/to/dlscripts/download.py [instagram] httpheader_Cookie = ... httpheader_User-Agent = ... no_livedl = false fail_if_not_following = true [instagram/home] count = 50

There are a lot more options available, these are just a few common/required ones. The README on Github gives a bit of an overview of how to use it, but here's the jist of it:

Each group inherits its parent groups in the path, and top-level groups inherit from the [default] group. So instagram/home is actually interpreted as:

[instagram/home] social_hooks = python3 /path/to/dlscripts/download.py httpheader_Cookie = ... httpheader_User-Agent = ... no_livedl = false fail_if_not_following = true count = 50

The social_hooks option allows dlscript's download.py to run after most feeds (any that support generating to the "social" format). Of course, you have to replace /path/to/dlscripts/download.py to the real path where dlscripts/download.py is located, and make sure to use the absolute path (i.e. don't use ~ or .. ).

httpheader_* options set an HTTP header for the request. You can set any header you want, but for Instagram, you only need to set the Cookie and the User-Agent headers, both of which you can find through inspecting network traffic in your browser. To do so:

Go to any Instagram profile (and make sure you're logged in)

Press F12 or right click->Inspect Element

Go to the network tab in the newly opened developer tools

Reload, and click on the first/blue request. If you scroll down, you should see "Request headers". Simply copy the values (minus the leading Cookie: or User-Agent: ) to rssit's config.ini

If no_livedl is set to true , lives will not be recorded. This isn't actually an RSSit feature, but configuration options are sent to download.py (and any other social hooks), which then interprets this option.

fail_if_not_following will return a fake HTTP error (490) if you are not following the user. This is useful if you're adding a popular person in Instagram, because fans often create a fake user account when the star changes their username.

count determines the amount of entries it returns. Don't set this too high, as it is both slower, and more likely to fail.

Whenever you update the configuration, you can simply go to http://localhost:8123/reload in order to reload the configuration, instead of restarting the entire server.

To see more options, go to http://localhost:8123/f/instagram, which will list all Instagram-specific options and endpoints.

dlscripts

Create a file named tokens.json beside download.py (i.e. in the dlscripts folder):

{ "prefix": "~/Pictures/social/", "windows": false, "thresh_processes": 10, "thresh_sleep_times": 600, "no_live_cleanup": false, }

prefix is where download.py will store the photos/livestreams. You'll want this to be an empty directory.

is where will store the photos/livestreams. You'll want this to be an empty directory. Unless you're using an NTFS-based filesystem (i.e. you're using Windows), set windows to false . It renames files to not use symbols that don't work on NTFS.

to . It renames files to not use symbols that don't work on NTFS. thresh_processes and thresh_sleep_times control the maximum amount of concurrent download.py processes, and how many times a waiting process should sleep (2 seconds each time) before running regardless, respectively. Note that download.py will stay running for any livestream that is being recorded/uploaded, so you might want to make this number higher (personally I have it at 15)

and control the maximum amount of concurrent processes, and how many times a waiting process should sleep (2 seconds each time) before running regardless, respectively. Note that will stay running for any livestream that is being recorded/uploaded, so you might want to make this number higher (personally I have it at 15) no_live_cleanup determines whether it should clean up temporary files after a livestream. Don't set this to true unless you know what you're doing, as the temporary files take up twice the size of each live.

We're done with configuration, for now at least.

Running the server

Make sure Redis and MongoDB are running. Next, simply run rssit:

cd rssit python3 rssit.py

And then webrssview in a separate terminal:

cd webrssview node webrssview.js

For the first run, wait for maybe 30 seconds to a minute (make sure Listening on 8765 is printed before). This will allow it to initialize the database. After waiting a bit, kill it, and start it again. When you load http://localhost:8765/ in your browser, you should see a folder called "root" at the top.

That's it for running, and those two commands are all you'll have to do when you reboot as well.

Now for the second part of the setup: Adding accounts, and the feed structure (optional).

In order to add an Instagram account, all you need to do is to copy the profile link to RSSit (http://localhost:8123), and it'll return the URL.

For example, let's take SNSD's Taeyeon. Her account is: https://www.instagram.com/taeyeon_ss/ . Pasting that into RSSit, and we get:

http://localhost:8123/f/instagram/u/taeyeon_ss

This is the feed URL. If you go to that URL now, it will take a few seconds to generate, and if everything is configured properly, it will return an XML feed, and the terminal running RSSit should show that it's downloading her latest posts. If it doesn't, then make sure to fix it before moving on.

The next step is to add everything into webrssview.

Using WebRSSView

To add a feed to webrssview, load webrssview in your browser (http://localhost:8765), right click on a folder (if it's your first run, then the only folder available will be "root"), then select "Add Feed".

You will be presented with a few options:

Name: The name of the feed displayed in the feeds list. This will be auto-generated if left blank

URL: The feed URL. Required (obviously)

Update Interval: The minimum amount of time between each update

Thread: Using different threads allows you to reload multiple feeds at once For example, if you have a thread named "instagram", and another named "twitter", then any feed under the "instagram" thread will be able to be reloaded at the same time as "twitter" feeds. This is a very important property if you have a large number of feeds.

Special Indicator: Adds an "indicator" to the title (usually a single character). Setting this to a character will make the unread badge red, as well as provide desktop notifications.



To categorize, you can use folders (right click -> Add Folder). Note that the last 3 options in Add Feed are also available in the Add Folder dialog. This is important, because all properties will cascade down, similar to RSSit's configuration model. For example, if you create a folder with a custom value for "Thread" set, all feeds and folders below it will inherit that value, unless they specify it themselves.

This allows you to organize a folder structure like this ( {} are the options for the feeds, not part of the name):

root { update interval: 30 } sns instagram { thread: instagram } snsd { update interval: 20 } taeyeon { special indicator: ! } exo { update interval: 10 } sehun twitter { thread: twitter, special indicator: T } snsd taeyeon { special indicator: null } …



Under this folder structure, sns/instagram/snsd/taeyeon would inherit instagram as its thread, an update interval of 20 minutes, and a special indicator of ! .

Similarly, sns/twitter/snsd/taeyeon would inherit twitter as its thread, have an update interval of 30 minutes, but it would not have a special indicator, as T is overwritten by null (which is a special keyword for this very purpose).

Hopefully you get an idea of the way it works.

Setup (2/3)

This is where you have to decide what you is your end goal with using these scripts.

If what you want is just to record lives or posts/stories for yourself, but don't care about anything more complex, such as automatically uploading to Youtube, hooking lives/posts into social media accounts, or whatever else you can think of, then nothing too special has to be done in terms of structure.

However, if you want the more complex functionality, make sure you read "Setup (3/3)" before adding any of these feeds.

As mentioned earlier, to add an account's feed to webrssview, paste the account's URL into RSSit (http://localhost:8123/), and it will return the feed URL. However, there are also a number of other endpoints:

reels_tray

This is where lives and stories are held, and thankfully Instagram gives us a very permissive rate limit for it (~7-8 seconds). Since livestreams contain the last 10 seconds of footage, setting the refresh rate to anything below 10 seconds should result in being able to record the entire stream.

By default, if you were to add http://localhost:8123/f/instagram/reels_tray, it would contain contain both lives and stories, which would both pollute the feed, and make each request quite slow due to needing to fetch a specified amount of stories per call.

The way I personally have it setup instead, is to have two different feeds: One for livestreams, the other for stories.

There are a few ways to set this up, the first being through query string parameters:

Lives only: http://localhost:8123/f/instagram/reelstray?lives=true&stories=false&usereelstraycache=false Stories only: http://localhost:8123/f/instagram/reelstray?lives=false&stories=true&usereelstraycache=true

Note the use of use_reelstray_cache . This parameter allows the stories feed to run based off the cached reels_tray API call from the lives feed, to avoid needlessly requesting it again.

However, the way I personally did it was through profiles. To explain how to use them, here is how to achieve the same configuration above using profiles instead:

Add the following to ~/.config/rssit/config.ini

[instagram/reels_tray] lives = true use_reelstray_cache = false stories = false [instagram/reels_tray@stories] lives = false use_reelstray_cache = true stories = true

And add the following feeds:

Lives only: http://localhost:8123/f/instagram/reelstray Stories only: http://localhost:8123/f@stories/instagram/reelstray

The advantage of this method is not only cleaner URLs, but also being able to change the configuration without modifying the feed URL in webrssview (which would clear all previous entries).

Profiles can also be used to use multiple accounts, by modifying httpheader_Cookie .

home

Feed URL: http://localhost:8123/f/instagram/home

This is your home feed, which contains posts from everyone you follow. Don't rely on this to provide you with every post, but it's a good way to have posts appear quicker if you have many accounts added to webrssview.

Personally I have it set to reload every 10 minutes, as anything less tends to get rate limited for some reason. I've also set it to return at least 50 posts, through the count option (like any other option, you can set it either via config.ini, or through query string parameters).

news

Feed URL: http://localhost:8123/f/instagram/news

I stopped using this as it ended up taking up more than a third of the entire database, but this contains every action the people you follow are currently doing, as provided by Instagram.

It can also take quite a while to initially load, as it needs to do an API query for every unknown user ID (for example, if someone you follow, follows someone you don't follow). However, it will cache the results, meaning future calls will not take as long.

inbox

Feed URL: http://localhost:8123/f/instagram/inbox

Your instagram inbox. Useful if you don't check your phone much (like me).

Setup (3/3)

This part is optional, it's only if you want to automatically upload lives to Youtube, or hooking custom behavior when new lives/posts are found.

The way this works is that a script called parse_feeds.js in khelpers will parse your feed structure setup in webrssview, and generate a flat list of "members" (people) from it.

If this seems like a bad idea, that's because it is. It works, yes, but unfortunately very inefficiently. As I said at the beginning of the guide, if you're serious about making a channel to record livestreams like mine, I'd highly recommend using a spare computer.

parse_feeds requires a fairly strict structure to work. Here's an example:

root sns instagram 배우 이준기 소녀시대 前 Jessica Jung/정수연/鄭秀妍 (제시카/Jessica) 김태연/金太軟 김태연 # 제로 이순규 (써니/Sunny) 소녀시대 블랙핑크 ลลิษา มโนบาล/라리사 마노반/Lalisa Manobal (리사) twitter 소녀시대 김태연



There are a number of things going on here:

Each site (in this example, instagram and twitter) can be specified anywhere. In this case, they are side by side, but as long as they are not contained within each other, it doesn't matter where they are in the feed structure. You will manually input the path later.

Wherever possible, Korean is entered. Romanizations are specified later (if needed)

The format for the feed titles are: "real name (stage name) # comment" Multiple names can be split with / The native name should always come first Korean names should always come before romanized names The reason for this is that the Korean localization for video titles always uses the first name specified In Jessica's case, it would be shown as "前 소녀시대 제시카 (Jessica Jung)" The English video title would instead be "Ex-Sonyeoshidae Jessica (Jessica Jung)" Don't worry about the romanization, we'll get to that in a moment For Korean names, if stage names are not specified, they are automatically generated by removing the first character (last name) Since 김태연 wasn't given a stage name, it will generate a stage name of 태연 Because of this, 소녀시대 (the group's account) would actually be romanized as "So Nyeoshidae" To fix this, nickname generation has to be turned off for it (this will also be explained in a moment) Romanization is automatically generated. In some cases, it's fine. In others, you might have to manually override it 리나 will return Lisa, as expected 제시카 will return Jeshika, which is why Jessica was specified after (although you could also override it) It can be a little hard to predict, so a bit of trial and error might be necessary Names that are not shown in the video title (such as their chinese names) are specified in the tags instead, which (in theory) help people to find the video. Comments are ignored, but are useful if you need to specify multiple accounts for one person In this case, # 제로 was used to specify zero.taeyeon 's account

前 is used to specify ex members.

There is only one "member" (profile) object per person, meaning all accounts are merged together As you can see for 김태연, the other two instances do not contain her chinese name, as all names are merged together, rendering it useless (although also harmless) to add her chinese name for the other accounts.



Now that you have your feeds properly organized, create a file named feeds.toml next to parse_feeds.js (i.e. in the khelpers folder):

[general] # These are a list of folders that are to not be considered groups # In this case, "배우" (actor) is used to categorize actors in a folder, # but this shouldn't be shown when uploading a livestream nogroups = [ "배우" ] # These folders will be entirely ignored. # Useful for specifying related accounts (in this case, friends) ignorefolders = ["친구"] # Default base tags to add to youtube videos. These are the ones I used defaulttags = [ "kpop", "k-pop", "live", "livestream", "replay", "라이브", "라방", "다시보기", "방송" ] # Where all of the lives/posts etc. are downloaded, # This should be set to the same as "prefix" in tokens.json (dlscripts) dldir = "~/Pictures/social" # Youtube cookie/user agent, you can find this the same way you found the Instagram ones youtube_cookie = ... youtube_ua = ... # Templates for the title (the format will be explained later) member_title_template = "{{m.title_yt}{%m.member_fullname ({{m.member_fullname})}{%cm ft. {[cm {%$!length<70 {%$i>0 , }{{$.t}{%$.m.member_fullname ({{$.m.member_fullname})}}}} {{site.livename}" member_title_template_kr = "{{m.title_kr_yt}{%m.member_fullname_kr ({{m.member_fullname_kr})}{%cm ft. {[cm {%$!length<70 {%$i>0 , }{{$.tk}{%$.m.member_fullname_kr ({{$.m.member_fullname_kr})}}}} {{site.livename_kr}" basic_title_template = "@{{username}{%c ft. {[c {%$!length<70 {%$i>0 , }@{{$}}}} {{site.livename}" basic_title_template_kr = "@{{username}{%c ft. {[c {%$!length<70 {%$i>0 , }@{{$}}}} {{site.livename_kr}" timestamp_template = " [{{time}]" timestamp_template_kr = "{{time}" member_description_template = "{{site.sitename}: {{a.link}{%c

{[c ft. https://www.instagram.com/{{$}/{%!$last

}}}{%e_e

{{e_e}}



{{general.english_disclaimer}{%a.upload_privacy==public



This live was automatically uploaded}" member_description_template_kr = "{{site.sitename_kr}: {{a.link}{%c

{[c ft. https://www.instagram.com/{{$}/{%!$last

}}}{%e_k

{{e_k}}



{{general.korean_disclaimer}{%a.upload_privacy==public



이 영상은 자동으로 올렸습니다}" basic_description_template = "{{site.sitename}: @{{username}{%c (ft. {[c @{{$}{%!$last , }})}" basic_description_template_kr = "{{site.sitename_kr}: @{{username}{%c (ft. {[c @{{$}{%!$last , }})}" english_disclaimer = "This video was uploaded for fans who have missed the broadcast or want to watch it again. If you are in this video and wish for it to be removed, let me know. after confirming, I will delete it as soon as possible." korean_disclaimer = "못 보시거나 다시 보고 싶어하시는 팬들 위해 이 영상 올렸어요. 영상에 나오시고 삭제하고 싶어하시면 연락해주세요 확인되면 최대한 빨리 삭제할 거예요." [instagram] # Used for the templates sitename = "Instagram" sitename_kr = "인스타그램" livename = "Instagram Live" livename_kr = "인스타라이브" # Path to where the feeds are located in webrssview path = ["sns", "instagram"] # This allows you to specify "categories", which are top-level folders below the path (in this case, root/sns/instagram) # to organize groups. For example: # * instagram # * Korean groups # * 소녀시대 # * 김태연 # * ... # * Japanese groups # * AKB48 # * ... categories = false # Additional tags for youtube videos tags = [ "instagram", "인스타그램", "인스타", "인스타라이브" ] # The same options that exist for instagram can be applied here as well [twitter] path = ["sns", "twitter"] categories = false # Romanizations # A string or an array can be specified. # The first element of the array is the one that will be used, # but the others will be added in tags [roman] "소녀시대" = ["SNSD", "Girls Generation"] "브라운아이드걸스" = "Brown Eyed Girls" # Nick names/stage names # This is mainly used for groups, to either disable nickname generation, # or to specify alternate Korean names for the groups [nicks] "소녀시대" = false # This disables nickname generation, # which will allow it to be romanized as "SNSD" (above), # instead of So Nyeoshidae "브아걸" = "브라운아이드걸스" # Finally, we get to specifying individual accounts ["instagram/@taeyeon_ss"] # The default upload privacy is "private", this will allow lives to automatically be uploaded upload_privacy = "public" # This allows you to add their lives automatically to a playlist # (you'll have to make your own and paste the ID here) playlist = "PLXacTUOhQ2Iuv-RPDvT4VSpNUYuKh6qpf" # By default, notifications are disabled (meaning, it will not show up under your subscribers' subscriptions) # This will allow notifications. Be aware that too many notifications # will result in losing a lot of subscribers. notify_yt = true # This allows you to specify other accounts they might have. # They can also be added under the feeds in webrssview, but in some cases, doing it this way might be easier. alt_accounts = ["youtube/UC5z2fxN6rs69cSyXur6X6Mg"] ["instagram/@jessica.syj"] # This allows you to set the "real name" for English-localized video titles eng_kr_name = "제시카" # You can also set romanizations, just like you can under [roman] # This allows you to set per-account romanizations, instead of globally "제시카" = "Jessica" ["instagram/@ryuserasera"] # In some cases, you might not want an automatic stage name. This option will use the full name instead # Note that it will still generate the name by assuming the first character is the last name. # For example: 류세라 would be shown as Ryu Sera use_fullname = true ["instagram/@official_lai_kuanlin"] # If the first character isn't the last name, you can manually specify the full name # using a romanization instead "라이관린" = "Lai Kuanlin" # use_fullname is required to display the full name use_fullname = true ["instagram/@official.hasungwoon"] # If someone is/was part of more than one group, you can specify other groups they were part of here # This will not affect the title, but it will add it to the tags alt_groups = ["핫샷"] # If you'd rather not have the group to show in the video title, this option allows you to hide it hide_group = true ["instagram/@tropical__official"] # You can also add extra tags to specific member accounts tags = ["Bambino", "밤비노"]

Yeah, it's a little complex haha. Hopefully the comments will help it to make sense.

The template format is very important to understand, as it defines what your video titles and description will be. However, it's also a little esoteric, to which the only explanation is: I'm an idiot.

Anything within {} will be interpreted by the engine. Anything else is left as-is. The character after { specifies the command, so here's a table:

{ reads a variable. For example: {{time} returns the contents of the variable time There are a number of variables defined: m is the member object, it has a number of properties, but here are the main ones: m.title_yt is the member's "title". For example, "Ex-SNSD Jessica" m.title_kr_yt is the korean title, such as "前 소녀디새 제시카" m.member_fullname is the member's real name, such as "Jessica Jung" m.member_fullname_kr is the korean real name a is the account a.link is the link for the account a.upload_privacy is the upload privacy you set (if applicable, if not it will be private ) site is the object for the site of the account. The properties here are the same as what you specified under the group for the site For example, for Instagram, site.sitename would return the "sitename" property under [instagram] (which is "Instagram", if you didn't change it) c is a list of "co-authors", containing only their usernames cm is a list of co-authors, that are also specified under the feeds (i.e. have a "member" object). This contains two variables, m and a , which are equivalent to their top-level counterparts. e_e and e_k are "extra" text (in English/Korean respectively), usually added by scripts like remcopyright.js . You'd usually want it to be at the top of the description $!length is the length of the string so far (useful for staying within youtube's title limits) Any variable that is not known, will be taken from feeds.toml . general.english_disclaimer will return a variable named "english_disclaimer" from the [general] section There is no recursion in variables. Meaning, "english_disclaimer" would be added as-is, without interpreting any special blocks.

reads a variable. For example: returns the contents of the variable % is a conditional. If it's true, it will evaluate the rest of the block, if not, it will skip the rest. If the conditional is variable name, it will be true if the variable has a value. For example, if time is 190101 , then {%time Yes} will return "Yes" If the conditional starts with ! , the rest of the conditional is inverted. So true = false, false = true. Using the previous example, {%!time Yes} will return nothing Normal c-like conditionals are supported: == means "equal to" != means "not equal to" > means "greater than" (can only be used for numbers) >= means "greater than or equal to" (ditto) < means "less than" (same) <= means "less than or equal to" (also)

is a conditional. If it's true, it will evaluate the rest of the block, if not, it will skip the rest. [ is for iterating through lists, kind of like a "while" loop. The variable after it specifies which variable to iterate over (i.e. {[c would iterate over the variable c ) Within the block, there are a few special variables $ is the current element $i is the current index $last is true if it's the last element For example, if c is ["taeyeon_ss", "zero.taeyeon"] , then: {[c {{$} } would return "taeyeon_ss zero.taeyeon " (notice the space after) {[c {%$i>0 , }{{$}} would return "taeyeon_ss, zero.taeyeon" {[c {{$}{%!$last , }} would also return "taeyeon_ss, zero.taeyeon"

is for iterating through lists, kind of like a "while" loop.

Uploading automatically to Youtube

Now that you have your feeds.toml properly setup, automatically uploading to Youtube is probably the reason why you went through all that effort. Thankfully, this isn't (too) hard, but it will require an API key from Google.

Acquiring this is a bit beyond the scope of this tutorial, but here is Google's official guide: https://developers.google.com/youtube/v3/getting-started

Once you have your API key, download client_secret.json from the Google developers console, set it in the khelpers directory, then run:

node youtubeul.js

This will request that you follow an authorization URL, and make sure to login with your YouTube channel, not your personal account (unless they are the same). Paste your code, and you're done the authorization setup!

Next, all you have to do is to hook it into dlscript's iglivedl.py, by adding the following line to tokens.json (in dlscripts):

"live_hook": ["node", "/path/to/khelpers/youtubeul.js"]

Make sure to replace it with the real (full) path to khelper's youtubeul.js, and also make sure that the file is a proper JSON file (commas after every entry, } being at the end of the file, etc).

After that, you should be good to go!

Maintaining the system

Now that the system is fully setup, there are a few things you might have to do to maintain it.

Errors fetching feeds for individual users

Any feed with an error will show up with red text. Clicking on it will give a more detailed description at the top of the page. For users, there are 3 common HTTP errors:

404

This means that the specified username does not exist. If it previously worked, this either means they have changed their username or have deleted their account.

If they have changed their username, to find their new username, there are a few ways. The easiest is usually to find a post that still exists (i.e. pictures still load) in the feed and click on it. The new username should show up.

However, there are cases where finding a post that still exists is difficult or impossible. In those cases, right click on the feed, select Info, copy the UID from the description, then paste it into this url (replacing 12345 with the UID you copied):

http://localhost:8123/f/instagram/uid/12345

This will return a feed, and you'll be able to find the username in the <link> tag.

If the user is a Korean celebrity, consider submitting their new username to their namu and/or nautiljon page to help others find them :)

490

If you set fail_if_not_following to be true, this means you're not following the user, which usually means the original user has changed their username (and someone has created a new account/renamed an existing account with the old username).

500

This could mean a number of things (as it means an error in RSSit), it's usually a temporary error, but it could also mean that the user has either deactivated their account, or blocked you.

Deleting old posts

If you keep this running long enough, you'll likely eventually end up running out of space. I hacked together a simple script (remlive.py in dlscripts) that removes old posts supporting both a whitelist, and separate thresholds for different post types (lives, stories, and regular posts).

Create a file named remlive.json beside remlive.py (i.e. in dlscripts):

{ "protected": [], "protected_nolive": [], "fast_nolive": [], "ultrafast_nolive": [], "noigtv": [] }

Each of the entries are lists of usernames (individual usernames can exist in multiple arrays):

protected users will not have anything deleted

users will not have anything deleted protected_nolive will, as the name suggests, have only lives deleted

will, as the name suggests, have only lives deleted fast_nolive will have lives deleted more quickly at a faster threshold than the default

will have lives deleted more quickly at a faster threshold than the default ultrafast_nolive has an even smaller threshold

has an even smaller threshold noigtv removes IGTV posts for the users listed

To change the actual threshold values, they are at the top of remlive.py .

If you run out of disk space, both MongoDB and Redis will fail, you will need to restart both services after you've cleared up some space.

Database backups

Although not specific to webrssview, creating backups of your database is always a good idea. Thankfully mongo makes this very easy:

mongodump -o backup1

This will create a backup folder in the current directory named backup1 . Restoring is similarly easy:

mongorestore backup1

Maintaining a live recording channel

Obviously these are just suggestions, you are of course free to do whatever you want. These are just things I've learned from running my channel.

Rotated videos

Youtube used to let you rotate videos in their video editor, but unfortunately they've removed this functionality. So I created a simple script to do the same.

You'll likely want to rename the uploaded video to add [unrotated] or something before, as it will simply run youtubeul.js on the rotated video, meaning the live number (e.g. "Taeyeon Instagram Live 2") will be incremented.

Usage is simply:

node rotatevid.js /path/to/local/live.mp4 rotation

Where rotation is one of 90 (counter-clockwise), -90 (clockwise), or 180 .

Copyright claims

Don't confuse this with copyright strikes. These are (usually) automatically made by YouTube's Content ID system, and will occasionally block your videos in certain (or all) countries. The vast majority of the time, it's because some song is playing in the background, sometimes it's barely even audible, but Content ID's system is definitely quite impressive on a technical standpoint.

YouTube has a system for removing the songs/videos that are blocked, but in my experience, it's not only slow, it also often doesn't even work. So I instead created a more automatic system that will parse the copyright claim page, then automatically remove the blocked material and reupload.

Usage is quite simple. First, reupload the original video to some website (I used Google Photos). Next, just use:

node remcopyright.js /path/to/local/live.mp4 'youtube url' 'link to backup of original video' [optional rotation]

For example:

node remcopyright.js ~/Pictures/social/instagram/taeyeon_ss/(2019-01-01T00:00:00)\ \[LIVE\]\.mp4 'http://youtube.com/watch?v=somevideoid' 'http://photos.goo.gl/some_id'

Like with rotatevid.js , you'll likely want to add some kind of prefix to your originally uploaded video to avoid the live being misnumbered.

Copyright strikes

These are the nasty ones. The "3 strikes you're out" ones. Don't panic, delete (don't set them private, actually delete them, private videos can get striked) every other video they might strike, but do NOT delete the videos they strike. If those videos are deleted, it's much trickier to get rid of the strike (they need to contact YouTube's support team).

As to what to do about the strike, that's up to you. The only advice I can really give is to be kind and honest whenever possible.

Possibly problematic lives

Through experiences with people asking for deletion requests, here are lives that could potentially pose a risk to the people in the live (i.e. please be careful when uploading these):

Walking home. People can (and unfortunately have) used this to track their location.

Drinking with friends. Some of these types of lives have unfortunately led to problems, so please be careful when sharing these

Composing/band sessions. There have been a few times where new songs were accidentally leaked this way

Deletion requests

I always double check by sending a message to their Instagram, unless I know for sure it's them (such as their youtube being linked by their Instagram, etc.). Some people are nice, some people aren't, my advice is to just stay nice regardless and to comply with everyone, as long as you know it's actually them.

My process is that once I have confirmed it is actually them, unless they specify they only want specific lives removed, I remove every live they have done, and add an entry to feeds.toml:

[instagram/@username] noupload = true

This will add NOUPLOAD to the video title of future lives (if you used title_yt and title_kr_yt in the template). Also, make sure that there are no duplicate entries in feeds.toml (otherwise parse_feeds will fail, meaning no lives will be uploaded), and obviously also make sure that the upload_privacy isn't "public".

If they only appear for part of the video, you can also cut out part of the video through YouTube's video editing tools. In my experience, it usually takes ~15-30 minutes for the video to be updated.

Closing

That's about it I think :) If there's anything missing or that you're confused about, please let me know. I tried to make the most detailed guide I could, but I'm sure there are things I have missed.

There are a few things I haven't touched in this guide (such as running the discord bot), as I expect very few people would be interested in them (and it would take a considerable amount of time to write). But if you're interested, contact me and try to I'll help you set it up.

I know this system isn't very elegant, and overall quite inefficient. I honestly don't expect many people to use it, but I wanted to make sure to explain how to use it, in case anyone did.