"Visual Twitter" programming project (web coding tips & tutorial)

"Visual Twitter" programming project (web coding tips & tutorial)

July 12, 2019 46 By Bernardo Ryan



hey tech late here today what we're going to be doing for you is building a pet project and you can follow along we're going to build visual Twitter it's essentially like a feed of images taken from the Twitter feed and this is a pretty cool neat project we're going to go through a number of API is going to be a lot of fun and I encourage you guys to follow along if you want to get some pet project work on to your resume just as good resume filler material this video by the way is sponsored by hosting er we're going to be using their services or checking out hosting or comm slash tech lead they'll get you all set up with domains websites all of that stuff they'll get your pet project hose it for you and online is a great value and you can follow along as we build this website make sure to use code the tech lead to get 15% off it is coffee time by the way and I thought I would do something a little bit of fun today actually this is Atlas coffee club I kind of looked around found a nice coffee of the Month Club tons of fun they send you a little postcard and then to tell you all about what the coffee is here like they'll say that this one has notes of hazelnut and brown sugar and so if you mind drinking the same coffee now I'm drinking every month they were being linked in the description and we just have some fun with this it's good alright so what we're going to be creating today is visual Twitter and we're going to just show you guys how you might put a service like this together now I want to give you a one key piece of insight here before we start which is that we're going to be gluing together a lot of pieces and it's funny because I used to consider myself like a glue engineer I would just take a lot of different pieces and glue them together and I thought well does that really make me like a programmer am i a software engineer if all I do is just hook up different pieces together I take API endpoints of services that are already built you know systems frameworks and just glue them together and then present it in UI and that's essentially what generalist software engineers do and it's actually what a huge perf software engineering is all about just gluing pieces together we're going to be using a number of micro services today to get ourselves set up so let's take a look it's going to be fun alright so the first thing we're going to do here is we're going to go over to host tinker com slash tech lead and we're going to get ourselves a website set up such that we can actually build this service out make sure to use code the tech lead to get 15% off since I already have a website I'm going to set up a sub-domain Twitter Tecla procom it's going to set this up with a private directory for this and once I have to set up I'll be able to FTP into it and then we'll be able to add code to it and build out our program I'm going to be using Visual Studio code you can use any FTP editor you want they have a plug-in called FTP simple and I'll be able to use is to configure it and connect in so now if I access Twitter Tecla procom then it will show me the site that I've created you know I'm just running PHP info and we're going to use PHP you can use really any language that you want but PHP is it's a pretty simple basic language if you want to use this to get this set up and the first thing we're going to do here is we're going to try to fetch the Twitter feed and just scrape that so if I look up San Francisco I can see what that URL is going to look like and I can just grab that so if I'll get contents and yeah there you go you can see pretty much I've cloned the Twitter website all I've done is I've scraped it and then I've output whatever it had there now there's a few issues with what's going on here with this grading process and things are problematic number one is we may get blacklisted they actually do have a developer API which you can purchase from them but you know it's kind of expensive and I'm not sure I want to buy that it turns out actually that there are services known as scrapers and their job their whole job is that they will scrape websites for you and they proxy your request through thousands of random servers that they have and they'll be able to just scrape websites for you and essentially prevent you from getting blocked these Graper you know they're essential they allow you access to data and your data is highly important especially when there is no API and you're able to get that data it allows you to create services unlock barriers to be able to launch things and who knows maybe later on when you do become successful you'll be able to actually afford to go buy that data make deals whatever it is that you need to do so if you're just out to build like proof of concept I would say like these scraper services they may be great actually as an interim solution so there is this website we're going to check it out it's called scraper API and I have a link for this in the description below as well if you're interested scraper API allows you to easily build scalable web scrapers it's an extremely simple API that they provide you just access a single URL API scraper API com you pass in your API key and then you tell them to URL that you want so we're going to just take this and plug it in alright looking good there so now we're scraping the Twitter website through prophesied URLs using this scraper API service you know the code looks really clean it's essentially a one-liner that just replaces that URL now what we're going to do is now that we have the data and we know that you know using the scraper service we're not going to get blocked we're going to use regular expressions just to rip out our the images in here alright so I spent a little time here and I came up with a regular expression which is able to extract the URLs and then the images so we'll be able to show the images and then want you to link to them the screen mapping so things are getting pretty interesting though I'm able to show an image I can tap on it and then it's going to show the actual post so that's kind of neat and I realized that we're not doing paging and one interesting thing is that the scraper API they seem like they may even be able to do some paging for you like just some JavaScript going and you can also dig deeper into this and get the additional URLs and fetch those as well I think Twitter is doing that through JavaScript and fetching as a API call through Jason I mean you can also just pay for the Twitter API but this is kind of for demo purposes now the last thing I want to show you here is that there's one more very interesting micro servers that we can be using and that is cloud image that I own so you may have noticed here that all these images are differently sized right and they're actually pretty big like I've manually resize these down to 300 pixels in width but in your reality they're pretty big and so what I really want to do is crop up each of these images to say 300 by 300 pixels right so we don't get all these spaces now normally the way you may have to do this is you have to use popular image resizing library known as image magic the command line is called convert and you'll be able to take an image fetch it resize it crop it applying the filters you want and then you have to host that image and they have to set up like a CDN content delivery network in order to serve that image quickly like you don't need the CDN if you don't want to put then images just load slowly and so a general rule of thumb is anytime you have image heavy assets or services that you want to be serving over the Internet and you want those to be loaded quickly for the end-user you're gonna want to use a CDN cloud the image the i/o and we'll have a link for that in the description as well as a micro servers pretty neat one actually that allows you to do all sorts of image conversions and filtering and cropping and resizing and then they host it on their own Citians for you so you don't have to do any of this image hosting by yourself let's take a look at what they've got here very similar to scraper API which we were using this is also extremely simple like that simple to use you essentially just give the URL you tell it what you want in the URL parameters and you pass in the URL of the image that you want and they would just generate a brand new image for you and then it will cache that image and you know they're fast they've got to set up through CBN's when they're print these images I'm going to pass it through a special URL through cloud image which will instruct it that I just want to crop it to 300 by 300 pixels so that pretty much takes care of it it's so sleek and now if I were to reload my web page all right this is looking good now you so you can see here that I have managed to just size all of these images to 300 by 300 pixels one simple command line and you know it's all set up I can click on these and it goes into the actual Twitter post that's pretty much like a visual Twitter that I have been able to set up so pretty fun stuff here and you know one more thing we can do here is we can actually add the search field and just make sure that we're searching for whatever people want to look for here yeah so that's interesting so see using this technique I'm able to look up the most recent images of like say sunsets that have been posted to Twitter which is that a feature that even Twitter is not really exposing right like you can look up pictures of sunsets but they're going to try and deliver for you the most relevant pictures but if you want to see up-to-date pictures well you know this kind of thing seems like it's doing it I can put in a search here you can check out the latest images from London right looks like people are having some ice cream over they're having some drinks and you know this really only scrapes just that first page which is why I'm searching the word pic which just increases the chance that I'm gonna get some images here and yeah you know I can see like seems like yet this was somebody going around Tokyo so kind of in retrospect I would say that actually the Twitter API maybe we should have just used the API and just paid for that because scraping the Twitter API feed is a little bit more difficult than I had anticipated essentially because they're messing around with JavaScript and stuff like that but you could host this on say hosting ur comm slash tech lead scrape it using the scraper API and if you have images you can use a micro surface like cloud image style to resize images get the hose on the CDN you know just format it make it fast performance and all of this you can be doing while you're drinking Atlas coffee from the Atlas coffee club while watching the tech lead so there we go that's a fun little project there's a few interesting things that I do want to note before we wrap up here and one is that scraping data is an amazing thing that I think people should learn how to do I think that especially as you're getting started being able to scrape some data use some of that stuff it's going to be a great way to build a pretty cool pretty fun prototype that's actually going to be able to get you some pretty impressive results as well actually you don't necessarily want to do this too much but as you're getting started it's just a great way to start building prototypes and get ideas and projects off the ground you know building things like these pet projects one of the best ways to fill out your resume and have good stories to talk about as you're going through interviews or even if you're not even if you just want to build some things for fun they'll do for me if you liked the video give it a like and subscribe and I'll see you next time thanks bye