[ERROR] CRITICAL:twint.run:Twint:Feed:noDataExpecting ~ Inconsistent results [High Severity]

Python 3.6
twint 2.1.7 updated from master
Have searched issues without finding anything
Running on Ubuntu 18.04, anaconda, jupyter notebook
Commands run:
import twint
import pandas as pd

c = twint.Config()
c.Pandas=True
c.Search = "#nfl"
c.Hide_output=True
c.Since = '2019-12-01'
c.Until = '2019-12-02'

twint.run.Search(c)
df = twint.storage.panda.Tweets_df

Hi, thanks for writing this package, it's very useful.  I'm clearly not using it right though.  I ran the commands above as a test, using"#nfl" as a query because it's innocuous and guaranteed to have a lot of results over the course of one day, but I am getting inconsistent results.
First, when I run it I get a lot of these warnings (which I saw from another issue are probably related to http/https?):  CRITICAL:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
That's fine though, the script still runs.  The problem is the results are inconsistent.  I ran it last night and got back 6,832 tweets, then ran it again this morning as part of testing some other code and got 4,710 tweets.  When I saw that I ran it again and got 0 tweets.
I have a couple of questions if that's okay.  Is twint caching the results of queries somewhere, and if so, how do I clear the cache?  Is this inconsistent behaviour expected (is it a Twitter search page thing?) and if so, does it make sense to run the same search multiple times and concatenate results?  Finally, is there a suggested best practice for searching date ranges?  (i.e. If you want all the tweets for a hashtag for the past 3 months, is it better to do one big search or break the search into daily or weekly time ranges?)
Again, thanks for this package.  Great work.

Twint does not cache results, queries or anything else. Every single piece of data is provided by Twitter. It makes sense to run multiple searches until it makes sense to you, for example you could monitor specific hashtags and see which user deletes more tweets than others. On the other side, you will most probably get duplicates if you don't filter the data with since and until (or other parameters).
If the hashtag is really popular, I'd split the searches in months if not even weeks.

Thanks.  So, is it normal though that if I run the exact same search for the same query 12 hours apart that I get very different numbers of tweets back?  When I ran it yesterday I got ~6800 results and when I ran it again this morning I only got ~4700.  (When I ran it a third time it returned 0 tweets??)
It's not really that big a deal, I'm just hoping to understand the expected behaviour to explain to analysts when I explain the data to them.  If I'm only getting a subset, or a random subset of the data, that's fine, but I need to understand that before they ask me about it.
Thanks

If you run the same script over time, so with the same date-time ranges and other parameters, the dataset should be "constant" since it does not depend by the relative time which you run the script at.
So if we have a variation and what can variate is the time interval or the dataset itself, I guess that (in this case) is the dataset that's changing. Most probably due to deleted tweets, accounts that go from open to closed, accounts that get delete/suspended.
Let's say that tweets with id = 1,2,3 are sent in the interval time (A,B). If we run the scripts are C with C > B, and the tweets don't get deleted/shadowed et similia , we'll always get tweets with id = 1,2,3. The interval is closed and nothing can go it or get out by itself. The only way which a tweet can go out of that interval is just being deleted/shadowed and so on.

Thank you.   I guess my strange results were a glitch. I mean, it's unlikely that 2000 tweets got deleted for that hashtag overnight.  I'll keep experimenting.
Much appreciated.

I can guarantee you that what Twint returns is what Twitter gives (and this can be proven, just run with --debug or config.Debug = True and in the twint-request_urls.log file you'll see every request made, and you can replicate the request with any software of your choice).
You can run the script with your handle as target, if you always get the same tweets everything is fine otherwise it needs to be investigated.
Best of luck!

Thanks, I'm sure it's something I'm doing.  I will keep at it.

Sorry to come back to this, but just wanted you to be aware of the results of my testing.
I ran this exact code multiple times in a row, and each time it returned very different results.  I don't know if there is something wrong with my code (please tell me if so!) or if there is something else going on.
(running with python 3.6 and fully updated twint)
c = twint.Config()
c.Hide_output=True
c.Pandas_clean=True
c.Pandas=True
c.Search="#nfl"
c.Since='2019-12-01'
c.Until='2019-12-02'

twint.run.Search(c)

I just ran that exact code 4 times in a row.  It returned this many tweets:
Run #1: 1,909 tweets
Run #2: 280 tweets
Run #3: 13,207 tweets
Run #4: 3,015 tweets
I'm really not sure what to do with that at this point?  Am I doing something wrong? On the three runs with the lower values, it also returned this ```
CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!
Any advice would be welcome.

I tried your query and I can confirm that the results are not consistent. That's really strange and needs to be investigated.
I'll keep you updated

Thanks!

So it seems that having HTTPS or not does not always have effects, for now my findings are there (I've run the script 3 times):
==> nfl1.csv <==
1201274724778565632,1201274724778565632,1575241187000,2019-12-01,23:59:47,CET,1180200593962369024,yotesglendale,GlendaleCardinals,,Brandin Cooks leaves Budda Baker in the dust💨 #NFL #NFLSunday #LARams #LAvsAZ #redsea pic.twitter.com/QowdS89R0S,[],[],[],0,0,0,"['#nfl', '#nflsunday', '#larams', '#lavsaz', '#redsea']",[],https://twitter.com/YotesGlendale/status/1201274724778565632,False,,1,,,,,,,"[{'user_id': '1180200593962369024', 'username': 'YotesGlendale'}]",,,,

==> nfl2.csv <==
1201274724778565632,1201274724778565632,1575241187000,2019-12-01,23:59:47,CET,1180200593962369024,yotesglendale,GlendaleCardinals,,Brandin Cooks leaves Budda Baker in the dust💨 #NFL #NFLSunday #LARams #LAvsAZ #redsea pic.twitter.com/QowdS89R0S,[],[],[],0,0,0,"['#nfl', '#nflsunday', '#larams', '#lavsaz', '#redsea']",[],https://twitter.com/YotesGlendale/status/1201274724778565632,False,,1,,,,,,,"[{'user_id': '1180200593962369024', 'username': 'YotesGlendale'}]",,,,

==> nfl3.csv <==
1201274724778565632,1201274724778565632,1575241187000,2019-12-01,23:59:47,CET,1180200593962369024,yotesglendale,GlendaleCardinals,,Brandin Cooks leaves Budda Baker in the dust💨 #NFL #NFLSunday #LARams #LAvsAZ #redsea pic.twitter.com/QowdS89R0S,[],[],[],0,0,0,"['#nfl', '#nflsunday', '#larams', '#lavsaz', '#redsea']",[],https://twitter.com/YotesGlendale/status/1201274724778565632,False,,1,,,,,,,"[{'user_id': '1180200593962369024', 'username': 'YotesGlendale'}]",,,,

So as we can see, Twint starts always at the same point. Which is good.
Now we have to see where it stops
==> nfl1.csv <==
1201248296645398528,1201248296645398528,1575234886000,2019-12-01,22:14:46,CET,178163508,randi_heatlifer,Randi Hilsercop,,The jets are so bad that the bengals just beat them 😂😂 #NYJvsCIN #NFL  pic.twitter.com/5OEHqX1I45,[],[],[],0,0,0,"['#nyjvscin', '#nfl']",[],https://twitter.com/Randi_heatlifer/status/1201248296645398528,False,,1,,,,,,,"[{'user_id': '178163508', 'username': 'Randi_heatlifer'}]",,,,

==> nfl2.csv <==
1200912390356783104,1200912390356783104,1575154800000,2019-12-01,00:00:00,CET,4059670933,blowoutbuzz,BlowoutBuzz,,YourDozen: Get your NFL Week 13 picks in to win 2017-19 sets >>  http://bit.ly/2Ok7pdQ  #collect @PaniniAmerica #TheHobby #NFL #predictions #picks pic.twitter.com/ZQFTduKvrh,['paniniamerica'],['http://bit.ly/2Ok7pdQ'],['https://pbs.twimg.com/media/EKkj3X2WsAALfcr.jpg'],0,0,0,"['#collect', '#thehobby', '#nfl', '#predictions', '#picks']",[],https://twitter.com/BlowoutBuzz/status/1200912390356783104,False,,0,,,,,,,"[{'user_id': '4059670933', 'username': 'BlowoutBuzz'}, {'user_id': '44128979', 'username': 'PaniniAmerica'}]",,,,

==> nfl3.csv <==
1201245479746646022,1201245479746646022,1575234215000,2019-12-01,22:03:35,CET,2691199254,seahawksreddit,/r/Seahawks,, https://ift.tt/2DyJ51V  Ravens Win! #Seahawks #NFL #GoHawks,[],['https://ift.tt/2DyJ51V'],[],0,1,6,"['#seahawks', '#nfl', '#gohawks']",[],https://twitter.com/SeahawksReddit/status/1201245479746646022,False,,0,,,,,,,"[{'user_id': '2691199254', 'username': 'SeahawksReddit'}]",,,,

If we take a look at the twint-last-request.log file (when Twint exists with error)

Sorry, that page doesn't exist

Back to home

      </div

If we take a look at the latest scraped tweet in nfl2.csv we see that its time is 00:00:00 (relative of my TZ) which is good, it says that we reached the end of "my day".
A note about the time-zone. If we run the same with two different local times, most probably we'll get different results since my start (end) of the day is different that yours. That said, our aim is not to be sure that one gets each other's results, our aim is to get the same results each time we ask for them, comparing results individually. (FYI I got 25282 tweets)
Reasons why the issue might be related to HTTP(s) switch:

since the switch we've had error messages
in the past we've tested HTTPS and got the error messages

Reasons why the issue might not be related to that switch:

the error messages are printed at first, then those vanish
for valid requests of every run (HTTP or not), the content (tweets) is always returned

What happens when Twint gets those error messages:

Twint changes the UserAgent
https://github.com/twintproject/twint/blob/3a4f778233257dd902f6557a38998bfcc3a046bc/twint/run.py#L89-L94
Twint re-runs the same request (since self.feed and self.init don't change, the request's params are still the same)

Sometimes, luckily, error messages are not printed, even if running the same query. In such case the only thing that Twint has different than in the other runs is the UserAgent.
So maybe Twitter plays differently based on the UserAgent specified.
Updates soon.

It seems that using 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36` as UserAgent (as suggested by @o7n in #587), allows us to get almost all the expected results
So I suggest you to edit lines 158 and 159 in url.py  and to specify that user-agent, or another one suggested. Then please run and let me know if you get consistent results @jomorrcode

Hmmm. My installed url.py only has 146 lines (installed twint 2.1.9 with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint as per  the instructions).
I looked through it and couldn't find any reference to user agent.  I found it several places in run.py, but I didn't want to just start randomly changing the code without knowing that I was doing.

Sorry I mean in get.py, my bad
https://github.com/twintproject/twint/blob/3a4f778233257dd902f6557a38998bfcc3a046bc/twint/get.py#L155-L161

Sorry, finally had a chance to try this.   Yes, I made that change with that user agent and ran the code 4 times, and got back 13,011 tweets each time.

Just to update, running with that user agent ususally seems to bring back a fairly complete list of tweets, but does randomly fail and a smaller subset are returned.  I am playing around with a workaround to look at the last tweet returned to see if it's timestamp is close to 00:00:00, and if not, redo the query.    Not sure if there's a more effective way to detect that the scrape finished early as a workaround.

It would help a lot if there was some way to know if there were any errors during the search or profile request. Now the only indication of something going badly is a message on stderr, if any.

It would help a lot if there was some way to know if there were any errors during the search or profile request. Now the only indication of something going badly is a message on stderr, if any.

I thought I'd just add that after a lot more experimenting, results continue to be inconsistent regardless of the User Agent, though quite randomly so.  Sometimes I can run the same code 3 times and get the exact same number of tweets and other times it will return a much smaller number, or even zero tweets.

And does in happen regardless the query?

Huh, oddly it does seem to vary with the query.  Some queries I tried seem to always return the same number, others vary a lot.  Really strange that.  That football hashtag (#nfl) is always very variable, but something like #france or #germany seems to be consistent.

That makes the debugging even harder, as of now I'd exclude a flaw in Twint. So I guess that somehow the issue is related to Twitter
It'd be interesting to check if it returns less tweets even if it reaches the end of the day. Because if it stops before for unknown reasons, you could just resume from that point.
To try this out just run something like twint -s "#nfl" --debug --resume "test_1.session" --since "2019-12-18" --until "2019-12-19" --csv -o "test_1.csv"
import twint

c = twint.Config()
c.Search = "#nfl"
c.Debug = True
c.Resume = "test_1.session"
c.Since = "2019-12-18"
c.Until = "2019-12-19"
c.Store_csv = True
c.Output = "test_1.csv"

twint.run.Search(c)

If it does not stop at 00:00:00, you can just re-run the script/command as is and it will resume from where it started. You might want to apply some more complex logging, but keeping track of session files would be enough. Please consider that at every run the session file is overwritten, so you would like to do something like python3 script.py && cat test_1.session >> history.sessions or twint .... >> history.session. So that we can compare the session ids with the debug files (twint-request_urls.log is not overwritten at every run)

Oh, that is perfect.  I will try that.  I was trying to do something like that myself by checking for whether the last tweet collected was close to 00:00:00, but when it wasn't I had to rerun the who script instead of just restarting from where it stopped.
So if I understand correctly, as long as the script hasn't terminated, re-running twint.run.search(c) will restart from the last tweet collected (assuming it hadn't reached 00:00:00), so a simple loop with a check for the latest time collected should do the trick.
Thanks, you've been very helpful with this.

by checking for whether the last tweet collected was close to 00:00:00

That is a bit too subjective for me. When is a tweet "close" to 00:00:00 ? And for search queries with a lower volume there is no visible difference between "no tweet exists" and "no tweet was received".

That is a bit too subjective for me. When is a tweet "close" to 00:00:00 ? And for search queries with a lower volume there is no visible difference between "no tweet exists" and "no tweet was received".

This is very true.  For what it's worth, because I was looking at fairly active search terms, I was considering it an incomplete search if no tweets were returned within 30 mins of 00:00:00, but as you say, that's not actually an effective way to handle this since it could easily happen that by chance there were no such tweets. I was including a cut out so that if the search ran 4 times without reaching 00:00:00 it would exit gracefully and print an alert.  Again, a hack to get around this that doesn't actually work that well.
I don't have a better idea at the moment though.

I do see that the file "twint-last-request.log" contains a field "has_more_items":true.  Is that something that can be accessed and used as a check before the script terminates to say, if has_more_items is true, re-run the search and resume from the last tweet collected?  I'm not sure where that information resides before it is written to the log file.

@o7n you are absolutely right, but with the sample cited above we suspect that the latest tweets is<PHONE_NUMBER>356783104,1200912390356783104,1575154800000,2019-12-01,00:00:00,CET. So considering this testing query (since with smaller ones I'm unable to replicate the issue) we can try to better understand what happens
As you said, and as we all agree, that's not a general rule. Instead it's a case-specific one

@jomorrcode so you mean like if at the request N we get has_more_items:true and at the request N+1 Twint breaks, Twint retries the query since we now that there are more tweets?
That sounds good, indeed when Twint breaks we lose the information about the previous request. Let me check what we can do

I create a new branch for this
https://github.com/twintproject/twint/tree/workaround-604
I'll push fixes there, so please remember to pull from that branch!

Added a new log file twint-requests-deep.csv, that contains rows formatted as follows f"had_more_items:{self._has_more_items};has_more_items:{self.has_more_items};init:{self.init};len_feed:{len(self.feed)}"
This will help us tracking down the requests.
If a large amount of tweets will be missing from a bigger set, we expect to see a difference in twint-requests-deep.csv

Hi, I did a couple of tests using both the Master branch and then the workaround branch, with a fresh install, I ran your query for "#nfl" for 2019-12-01 to 2019-12-02.  I made sure to uninstall the Master branch before installing the workaround branch.
Results for 4 runs with Master Branch:
Test 1: time= 1 min, total tweets = 680, last tweet =<PHONE_NUMBER>531468800 @ 22:47:13
Multiple errors:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1
Test 2: time= 10 sec,  total tweets = 0
No errors returned at all
Test 3: time= 11 mins, total tweets = 13,074, last tweet =<PHONE_NUMBER>342723073 @ 00:00:05
A few errors:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
Test 4: time= 1 min, total tweets = 100, last tweet =<PHONE_NUMBER>176659969 @ 23:45:39
Multiple errors:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1
Results for 4 runs with Workaround branch
Test 5: time = 14 mins, total tweets = 13,074, last tweet =<PHONE_NUMBER>342723073 @ 00:00:05
Dozens of errors
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1
Test 6-8: basically identical to #5 (between 13,070 and 13,080 tweets, all ending with the same last tweet)
If it's useful I save each of the session, request logs, and deep_test files for each of those, as well as the actual tweets csvs.

Downgraded to the version 2.1.6 and it's working !

Thanks @Aassifh for notifying us, I'll do some tests!

how did u do downgrade?

pip install twint==2.1.6

i install 2 and works finally ... the problem is the last version with elasticsearch so ...

I was facing this issue when I travelled from Boulder to India. I was constantly shown the warning/error - 'CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)' and the output was inconsistent. I tried many different stuff, but I kept facing this error in India. When I was back in Boulder, I ran the same piece of code again, and guess what? It ran without any errors!
So I feel that at least in my case, it was some regional setting of the network w.r.t accessing Twitter data which was causing this error, I can't say in detail though.

Maybe its has something to do with the timezone ! check it !

It sounds like you guys are battling a machine learning process that's trying to detect scraping.

I'm starting to think the same thing. @edsu.

I would not exclude that, I think that's totally possible

This feels more like simple rate limiting or capacity management mechanisms.

Might be that the detector uses a specific algorithm, that may run in a ML process or not, and the effect is rate limiting

And how would it be more effective than just rate limiting on its own? This is all just speculation, there is not a single piece of evidence for a ML process.

@o7n actually nobody said for sure that behind this issue there's a ML process. We are just sharing our thoughts

I’m just sharing my thoughts as well.

I know its a fresh issue, but any ideas for a solution? I've tried downgrading Twint version and it didn't work. When I can scrape between 8-10MB of raw tweets to CSV file it displays the discussed error. PS: awesome tool!

I used to see this message while collecting tweets:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
But the messages were showing up intermittently and the process continued with no issues. Now, the process stops after receiving the first message while sending a sequence of the above messages. Then, for ~1 minute the message keep showing up and after that Twint successfully can collect tweets like before getting those messages again.
I wonder if we can hibernate the process for a few seconds and start over the process after. Is it possible? I didn't look into the codes yet.

As a temporary solution, I tried this and then the collector did not fell off. I change the part of the twint code in run.py before getting a random UserAgent.
                if consecutive_errors_count < self.config.Retries_count:
                    #################################
                    delay = random.randint(60, 120)
                    print('sleeping for {} secs'.format(delay))
                    time.sleep(delay)
                    #################################
                    self.user_agent = await get.RandomUserAgent()

                    continue
                logme.critical(__name__+':Twint:Feed:Tweets_known_error:' + str(e))
                print(str(e) + " [x] run.Feed")
                print("[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!")
                break

It seems the collector is able to start over from the place it left off after 2 min. I know this is totally inefficient. So, please share your better solution to counteract the
Twitter limitations.

So, please share your better solution to counteract the
Twitter limitations.

I've also run into this error for larger scrapes. If it's, in fact, an issue with rate-limiting from twitter, what would it take to implement a config option that slows down requests by a set # of seconds (e.g., new tweet is scraped every 3 seconds)?

How about a binary exponential drawback? Every time it fails it doubles the sleep time.
            if consecutive_errors_count < self.config.Retries_count:
                #################################
                delay = exp(2, consecutive_errors_count)
                print('sleeping for {} secs'.format(delay))
                time.sleep(delay)
                #################################
                self.user_agent = await get.RandomUserAgent()
                continue

By the way, can we make those errors print to stderr instead of stdout? It would make handling errors so much easier.

How about a binary exponential drawback? Every time it fails it doubles the sleep time.
Sounds good.

How about a binary exponential backoff? Every time it fails it doubles the sleep time.
            if consecutive_errors_count < self.config.Retries_count:
                #################################
                delay = exp(2, consecutive_errors_count)
                print('sleeping for {} secs'.format(delay))
                time.sleep(delay)
                #################################
                self.user_agent = await get.RandomUserAgent()
                continue

I like it. But for those of us regularly scraping large numbers, it'd be nice to have a pre-set delay that doesn't require an error to trigger. I could implement that manually in my script (iterate backwards by day or week with a pause between every item in the range), but it'd be more elegant to have a delay built-in via config option.

Yeah, I agree, but I think those are two different features.

I made a pull request here https://github.com/twintproject/twint/pull/686 with features for configurable delays and exponential backoff with a configurable base. Also a bug fix where after the first error the user agent would become random again, making things worse, and errors to stdout.

#684

I am scraping a lot of data each run per day around a large area and #686 did not solve my issue. It would not restart once i hit the limit; so instead I added along with it
                    elif self.config.TwitterSearch:
                    self.feed, self.init = feed.Json(response)
                    if self.config.Max_delay:
                        delay = random.randint(self.config.Min_delay, self.config.Max_delay)
                        sys.stderr.write('sleeping for {} secs\n'.format(delay))
                        time.sleep(delay)

Depending on the type of scraping the user needs a different type of delay timer. This has worked for me with the Min_delay = 0 and Max_delay=3. We can probably make this sample from a  normal distribution shifted towards 0 if you need it to be faster.

I found another workaround for this problem. It seems that Twitter block many requests from api. When we use TOR proxy ip address change every 10 minutes.

I'm also facing this issue. I think it may occur some rating limit by Twitter Server. However, I tried to set up a daily interval search in my code and run it every 1/2/3/4 minutes. In practice, this issue does not occur at 4-minute intervals. So, I think maybe this rating limitation has a specific time range. For example, the issue occurs if there are too many access requests within three minutes.

Yes, that is how rate limiting works. It kicks in when the number of requests over a sliding time interval (i.e. the last x minutes) is higher than a certain limit.

Hi there
I have been suffering these noDataExpecting issues.
Previously I was using CLI twint and it was great - then it started failing so I started looking at using python to see if there were any workarounds - but I hit the errors again and then found this thread
I am a developer but new to python (and driving twint from python) and Git (!!)
That said, how can I download your latest changes for this bug to see if they fix my issue?
Apologies if this is the wrong place to post this - (I did say I'm new to Git right?)
If appropriate feel free to delete my post and reply to me as best

Just throwing this in if it helps. I've identified that twitter blocks your IP after you've scraped roughly 15000 tweets, which resets after a period of 7 minutes. A simple workaround I did was generate a list of days I'm interested in scraping over and run the twint search function for each day. After each iteration, time.sleep(600) <- just to be on the safer side, you can go with 420 if you're feeling daring.

As per my experience, this error is due to the queries limited per time window of twitter. In my case, trying after a while works well. Can developers put the incremental delay to restart the job? Apart from this two naive questions are, where is the location of logs? and how to throw an exception for this bug.

@markowanga can you post the code? Also scrapping a lot each day and running into this problem.

Hey all, I face the same issue.
I  parametrized 120 jobs for 120 Dataframes with different user information to speed up my scraping process. For Friend- and Followerlists there no problems. But for Timeline Data I have these issues.
I would love to throw an exception for this error.
I think @patel-zeel is right and queries are limited. If I run just one job, it seems that it should be fine for twitter. But strange why it is just a problem for timelines.
However. I love your project! Thanks!

Just throwing this in if it helps. I've identified that twitter blocks your IP after you've scraped roughly 15000 tweets, which resets after a period of 7 minutes. A simple workaround I did was generate a list of days I'm interested in scraping over and run the twint search function for each day. After each iteration, time.sleep(600) <- just to be on the safer side, you can go with 420 if you're feeling daring.
It will definitely take a longer time for the script to run but as long as you're happy leaving the PC running it works!

You're right. But does someone know why this is only the case for Timelines?

For those looking to throw an exception to catch this, the place to do that appears to be in twint's run.py file.    Approximately line 102, right after it prints the "[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!" message.
Here's what I have plugged in there:
logme.critical(__name__ + ':Twint:Feed:Tweets_known_error:' + str(e))
print(str(e) + " [x] run.Feed")
print("[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!")

# my exception
raise Exception("Hit Rate Limit")

break # with the addition of the exception, this break is probably unnecessary

I catch that exception, sleep for 420 seconds, and then run twint.run.Search(c) again to pick up from the resume file.

Just in case it's of interest to anyone, here is the code I am using to scrape Covid data from January, looking for early signs.  With the added exception above being caught, I have it iterating day by day through the date range and it seems to work fairly flawlessly.  I don't pretend to be a software engineer, so take it for what it is.
import pandas as pd
import twint
from datetime import datetime, timedelta
from time import sleep
import os

query = 'coronavirus OR virus OR 2019-nCoV OR wuhan OR #WHO OR flu OR pneumonia'

start_str = "2020-01-01"
end_str = "2020-02-01"
start_date = pd.to_datetime(start_str, format='%Y-%m-%d', errors='ignore')
end_date = pd.to_datetime(end_str, format='%Y-%m-%d', errors='ignore')
data_folder = "D:/Data/covid_tweets/"
filename = f"{data_folder}covid_tweets_{start_str}_{end_str}.csv"
resume_file = f"{data_folder}resume.txt"

c = twint.Config()
c.Hide_output = False
c.Store_csv = True
c.Output = filename
c.Resume = resume_file
c.Search = query
c.Lang = 'en'

while start_date < end_date:

    check = 0
    c.Since = datetime.strftime(start_date, format='%Y-%m-%d')
    c.Until = datetime.strftime(start_date + timedelta(days=1), format='%Y-%m-%d')

    while check < 1:
        try:
            print("Running Search: Check ", start_date)
            twint.run.Search(c)
            check += 1

        except Exception as e:
            # pause when twitter blocks further scraping
            print(e, "Sleeping for 7 mins")
            print("Check: ", check)
            sleep(420)

    # before iterating to the next day, remove the resume file
    os.remove(resume_file)

    # increment the start date by one day
    start_date = start_date + timedelta(days=1)

Just in case it's of interest to anyone, here is the code I am using to scrape Covid data from January, looking for early signs. With the added exception above being caught, I have it iterating day by day through the date range and it seems to work fairly flawlessly. I don't pretend to be a software engineer, so take it for what it is.
import pandas as pd
import twint
from datetime import datetime, timedelta
from time import sleep
import os

query = 'coronavirus OR virus OR 2019-nCoV OR wuhan OR #WHO OR flu OR pneumonia'

start_str = "2020-01-01"
end_str = "2020-02-01"
start_date = pd.to_datetime(start_str, format='%Y-%m-%d', errors='ignore')
end_date = pd.to_datetime(end_str, format='%Y-%m-%d', errors='ignore')
data_folder = "D:/Data/covid_tweets/"
filename = f"{data_folder}covid_tweets_{start_str}_{end_str}.csv"
resume_file = f"{data_folder}resume.txt"

c = twint.Config()
c.Hide_output = False
c.Store_csv = True
c.Output = filename
c.Resume = resume_file
c.Search = query
c.Lang = 'en'

while start_date < end_date:

    check = 0
    c.Since = datetime.strftime(start_date, format='%Y-%m-%d')
    c.Until = datetime.strftime(start_date + timedelta(days=1), format='%Y-%m-%d')

    while check < 1:
        try:
            print("Running Search: Check ", start_date)
            twint.run.Search(c)
            check += 1

        except Exception as e:
            # pause when twitter blocks further scraping
            print(e, "Sleeping for 7 mins")
            print("Check: ", check)
            sleep(420)

    # before iterating to the next day, remove the resume file
    os.remove(resume_file)

    # increment the start date by one day
    start_date = start_date + timedelta(days=1)

Thank you for sharing this. As for the delay, 2~4 mins works for me.
But I wonder how long does it take roughly to collect tweet for one single day for you?

I find it totally depends upon the volume of tweets for your search terms.  This morning I was looking at some covid19 keywords for 1 day in late March and it was taking over an hour to get one hour worth of tweets just because the volume was extreme. The volume of tweets that had been sent during that hour was over 1000 per minute!
Obviously a less active search topic will be faster.  The search I posted above has so far taken about 5 hours to get the first 25 days of January.

Hey all, I face the same issue.
I parametrized 120 jobs for 120 Dataframes with different user information to speed up my scrapping process. For Friend- and Followerlists there no problems. But for Timeline Data I have these issues.
Would you clarify more, did you run 120 search jobs in parallel using Twint? Did you try to search for tweets using keywords as well? And if so how if works? I didn't try more than 5 scrapings in parallel.

I would guess that running many processes in parallel from the same IP address would greatly increase your chances of being banned, or at the very least would slam into the rate limit very quickly.  Just a guess though.

The search I posted above has so far taken about 5 hours to get the first 25 days of January.

I wonder why for me it works much slower. Even with some narrow-down conditions like Min_likes or  Min_replies, it takes ~100 orders of magnitude more time.

Hey all, I face the same issue.
I parametrized 120 jobs for 120 Dataframes with different user information to speed up my scrapping process. For Friend- and Followerlists there no problems. But for Timeline Data I have these issues.

Would you clarify more, did you run 120 search jobs in parallel using Twint? Did you try to search for tweets using keywords as well? And if so how if works? I didn't try more than 5 scrapings in parallel.

I scrape Tweets, Friends and Followers for more than 30k users via Twint. I parametrized via bash.
for l in $(seq 0 $batchlenght $END); do
python3 friends_data_pipeline.py  &
python3 follower_data_pipeline.py &
python3 timeline_data_pipeline.py  &
done
code snipped
`def get_timeline(self):
    if self.disable_twint_logger:
        os.environ["TWINT_DEBUG"] = "debug"

    if self.batch:
        print("\n" + "Start Fetching Timeline Data Batchwise")
        self.batch = int(self.batch)
        print("batch_data/batch_{}.json".format(self.batch))
        new_df = pd.read_json("batch_data/batch_{}.json".format(self.batch), orient="records", lines=True,
                              convert_dates=False)

        if self.reversed_it:
            print("Reverse DataFrame")
            new_df = reversed(new_df["user_id"])
        else:
            new_df = new_df["user_id"]

        while True:
            logging.info("\n" + "Timeline search" + "\n")
            for account_id in tqdm(new_df):
                timeline_path = os.path.join(self.TIMELINE_DIR, str("{}_timeline").format(account_id) + '.json')
                if os.path.exists(timeline_path):
                    print("exist - pass")
                    continue
                else:
                    time.sleep(10)
                    timeline = twint.Config()
                    timeline.Limit = int(self.limit)
                    timeline.Hide_output = True
                    timeline.User_id = str(account_id)
                    timeline.Output = timeline_path
                    timeline.Store_json = True
                    try:
                        twint.run.Search(timeline)
                    except:
                        print("\n" + str(account_id))
                        print("Having that error because that account is suspended or protected" + "\n")
                        continue`

I only have problems with the timelines. Friends & Followers worked fine.

I would guess that running many processes in parallel from the same IP address would greatly increase your chances of being banned, or at the very least would slam into the rate limit very quickly. Just a guess though.

Yes, I think that's an issue. At least for the timelines...
Unfortunately, I´m not familiar with proxy / VPN implementations in such scraping frameworks.
I know that twint has a proxy functionality and my gut feeling says to me that this could be a good workaround. What about using a proxy pool? Does someone have experience?

I would guess that running many processes in parallel from the same IP address would greatly increase your chances of being banned, or at the very least would slam into the rate limit very quickly. Just a guess though.

Yes, I think that's an issue. At least for the timelines...
Unfortunately, I´m not familiar with proxy / VPN implementations in such scraping frameworks.
I know that twint has a proxy functionality and my gut feeling says to me that this could be a good workaround. What about using a proxy pool? Does someone have experience?

Sorry for the spam.
How about to scrape  https://www.socks-proxy.net/ to get updated ip adresses.

Judging by the rate of tweets collected,
Using a large time break like 420 or 300 is much slower (more than 2x slower) than just using a random sleep timer between 1-10 seconds in between each query. With a random sleep interval you can also run multiple twint programs at once also. I run two in paralell but it could work with more as I have yet to test its limits.

Judging by the rate of tweets collected,
Using a large time break like 420 or 300 is much slower (more than 2x slower) than just using a random sleep timer between 1-10 seconds in between each query. With a random sleep interval you can also run multiple twint programs at once also. I run two in paralell but it could work with more as I have yet to test its limits.

I run a test for 1 and 4 jobs and hit the rate limits twice. But not so fast.
However, I raise an exception and use a random sleep timer between 1-10 seconds. Wish me luck. 19 k users left :)

@give-me-data Hope this works well because if so then this would definitely be the fastest method up to this point! good luck!

For all who are interested why you hit the rate limits for timeline search but not for friends and follower search. I aksed the developer and he said:

so friends and followers use the "no-js version" of Twitter, while the timeline uses another endpoint.

Judging by the rate of tweets collected,
Using a large time break like 420 or 300 is much slower (more than 2x slower) than just using a random sleep timer between 1-10 seconds in between each query. With a random sleep interval you can also run multiple twint programs at once also. I run two in paralell but it could work with more as I have yet to test its limits.

I run a test for 1 and 4 jobs and hit the rate limits twice. But not so fast.
However, I raise an exception (run.py) and use a random sleep timer between 1-10 seconds. Wish me luck. 19 k users left :)

Could you please elaborate a bit more on how you "raise an exception (run.py) and use a random sleep timer between 1-10 seconds."? I am stuck and am having a hard time getting this to work.

run twint.run.Search(c) again to pick up from the resume file.

Could you please share the piece of code when you run twint.run.Search(c) again to pick up from the resume file? Thanks!

My two above posts with code on April 15 show where I put the exception in run.py, and then the code where I am actually scraping (with a while loop to re-run twint.run.Search(c) when the exception is raised).
Note that while I used a long delay of 420 seconds, lots of other people are saying you can use a much shorter one.  I haven't had a chance to try that yet, but it would make things a lot faster.

Thanks for sharing your code! Do you think this will be provided as an option (also for the command line) in the next twint release? I have always used it from the CL and I currently don't have the time to set this up like you did.

Just an update, I've tried my code with varying lengths of sleep, and anything under 420 seconds seems unreliable.  I like the idea of the exponential backoff that other people have talked about so I'm going to look at that.
Adding a random 1-10 seconds between each query seems logically like it should take a lot longer though, no?   If my code hits the rate limit and sleeps every 15,000 tweets for 420 seconds, that still a lot shorter than the 15,000-150,000 seconds the random delay is adding to the scrape of those same 15,000 tweets.
Or have I misunderstood the random idea?

@jomorrcode actually you are right, I was under the impression that for each query the result returns 100 tweets (like in the twitter API) but after looking at it again, it is only 20. Since on avg. 1-10 second random delay is 5 seconds then we can say within 420 seconds we will approx, get
20(tweets per query) * (420/5) = ~1680, much less than scraping with the 420 second timer.
I believe adding an exponential timer is unnecessary since it would just force us to wait until at least 420 seconds have passed regardless.

Ah, I missed the part where the delay got added after 20 tweets.  I was thinking one, but as you say, even so it would be slower.  It might feel faster too because the scrape never actually pauses ;).

Yea plus my init assumption was wrong haha, but for now, the 420 wait limit should be implemented as the standard. Suggest a PR maybe?

That would be a ridiculous idea. Using a hard coded timer won't work if there are multiple processes on the same IP address. Also, that 420 seconds value has been found using trial and error and Twitter can change it at any time.
There is already a PR which I made which has an automatic, dynamic, exponential backoff (https://github.com/twintproject/twint/pull/686) but which is apparently being ignored.

A hardcoded timer is definitely not the answer, it should just be an adjustable parameter. With the default value of 420. If twitter changes it then simply adjusting the default is fine.
The exponential backoff is definitely a more sustainable solution though.

I tried to code a way to use a different proxy for each request but failed because of connection or proxy issues. Dont Know...

That would be a ridiculous idea.

Jeez man, no one thinks a hard coded timer is the ideal situation, but no need to be rude about it.  I saw your PR and thought it was the best idea.  Seems to be await the resolution of conflict though?

I just re-read the code changes for @o7n 's PR.  It's very good.  I like the solution very much.

Hi all
honestly I can't check/try/evaluate PR. I'm not ignoring your requests, comments and stuff. Simply I do not have the time for this. I'm really sorry.
If someone wants write rights, just say and I'll add him/her as collaborator
Hope you will understand my position.
Thank you,
Francesco

@pielco11 If it is possible, can I be added? I am going to be using this project for a while for work and will make improvements where and when I can.

Thanks a lot! So if I understand this correctly, thanks to the changes committed above it's now possible to add a parameter to the command line execution to avoid getting the error, right? And it seems there is a (semi-)automatic option there, which would be great.
Could you give me an example of how to use this effectively via the command line? What would I have to add to a normal command like twint -s 'QUERY' --since 2019-01-01 --until 2019-12-31 -o 'QUERY.csv' --csv --hashtags --count -l en? Help would be much appreciated :)

@wuqui the error is unavoidable when you are scraping a large amount. The only thing we can do is ensure the smallest wait times once Twitter blocks our IP for a certain amount of time. All this commit does is provide an easy way to find the most optimal time to wait for.
In the case Twtter changes the time your IP is blocked for, the dual timer will find good value to estimate it automatically. I suggest using --min-wait-time 50 and setting --backoff-exponent 3.0. These values seem to work best under any conditions regardless of when twitter changes it.

Hi, I am facing a related but more serious problem. When I run the script with the command "twint -s pineapple", I faced the "noDataExpecting value" after scraping 1-3days of tweets with multiple trials.  I tried other keywords or even with another ip but the same problem persist, I can only scrape tweets for few days duration and then encounter the "noDataExpecting value".
I am using the latest twint, python 3.6.5 and ubuntu 19.04, just wondering if you guys encounter this problem as well.
This is the latest output:
1259483985215807494 2020-05-10 22:02:36 HKT  Perhaps the soft right
1259483982204461056 2020-05-10 22:02:36 HKT <Warhorse_90> Pineapples can go. For a very SPECIFIC reason. 😈😈😈😈  https://twitter.com/Mawpy/status/1259237431804661760 …
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

https://twitter.com/ZusorOW/status/1258885451055800320
This is going to be a huge issue for Twint, especially if Twitter will remove the endpoint (almost sure)

https://twitter.com/ZusorOW/status/1258885451055800320
This is going to be a huge issue for Twint, especially if Twitter will remove the endpoint (almost sure)

@pielco11 can you elaborate a bit? I am assuming that twint relies on this endpoint?

@mr-devs correct
Twint is "exploiting" the legacy version of Twitter

Here's something I just learned...  Twitter is smart. They aren't just doing the rate limiting on individual ips.  I spun up 4 VMs on a cloud provider with 4 different IPs (but all from the same group) and Twitter simultaneously rate limited the 4 scrapers running on ips x.x.x.a, x.x.x.b, x.x.x.c, and x.x.x.d,
Clever folks.

For twitter and Instagram this is true. For rotating proxies try not to
have the end numbers be the close to each other! Good observation for
people trying proxies on these sites.
On Tue, May 19, 2020 at 5:34 AM John Morrow<EMAIL_ADDRESS>wrote:

Here's something I just learned... Twitter is smart. They aren't just
doing the rate limiting on individual ips. I spun up 4 VMs on a cloud
provider with 4 different IPs (but all from the same group) and Twitter
simultaneously rate limited the 4 scrapers running on ips x.x.x.a, x.x.x.b,
x.x.x.c, and x.x.x.d,
Clever folks.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/twintproject/twint/issues/604#issuecomment-630787888,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGM26KF32CCXHUXPQVWXPHDRSJ4EZANCNFSM4JZ4KC6A
.

I've hear that using AWS Lambdas "serverless" service with Twint is an effective workaround for most of the problems that we are experiencing. Has anyone used this? I am in the process of setting this up myself now (its a bit more complicated than just running a script) but I was curious if anyone has experience with this?

That is effective, but only as long as not too many people are doing it.

https://twitter.com/ZusorOW/status/1258885451055800320
This is going to be a huge issue for Twint, especially if Twitter will remove the endpoint (almost sure)

Does anyone know a path forward for resolving this? Which parts (if not all) of Twint will be impacted by this change?

I just tried and Twint still works

I was running something overnight and woke up to find it still running, so far so good.

Can someone verify that the functionality is indeed removed already per the announcement?

This is weird, everything broke for me two days ago? No matter what I try and scrape (tweets, followers, lookup data) I get "TypeError: 'NoneType' object is not subscriptable"

As of now it doesn't work as expected, but it's still possible to run searches with the mobile version
https://mobile.twitter.com/search?q=from%3Anoneprivacy&s=typd&x=0&y=0

Hi all, I am on:
Mac OS 10.15.5
Python 3.7.7
twint 2.1.20
I am looping over the script below for a range of days and saving to CSV after each search (I can post my full code if requested).
c = twint.Config()
c.Lang = 'en'
c.Search = '(mask OR masks OR facemask OR facemasks)'
c.Since = '2020-01-01'
c.Until = '2020-01-02'
c.Pandas = True
c.Hide_output = True
twint.run.Search(c)

I keep getting this repeatedly:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 1.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 8.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 64.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 125.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 216.0 secs

Am I correct that it keeps getting hung up on the same tweet and increasing the sleep time, until finally giving up after the 216 sec sleep?  Due to the amount of tweets I am hoping to scrape for this project (Jan 1 - present), I would prefer to skip over the tweet if it doesn't return after the 8 or 27 sec sleep.  Is there a twint method to do this?
I read through much of this thread as well as the others linked and did not see a direct answer.  Thanks in advance!

It's not being caused by any tweet in particular.  Twitter is blocking your IP for a fixed period of time (about 7 mins) after scraping a large batch of tweets.   This progressive backoff is the best work around we've found so far.

Gotcha, thanks for the FAST reply!  Loving twint by the way... I will keep monitoring this issue.

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1 column 1 (char 0) Expecting value: line 1 column 1 (char 0) [x] run.Feed [!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!
Receiving this error continuously and program exits after that, I have read the entire thread but wasn't able to find the solution. Could anyone guide me?
My code:
`
for city in PROVINCE_CITIES[province]:
print(city)
c = twint.Config()
        c.Search = keywords
        c.Since = since
        c.Store_csv = True
        c.Output = "./" + outfile

        c.Near = city
        c.Hide_output = True

        twint.run.Search(c)

`

looks like it started again -_-

looks like it started again -_-

looks like it started again -_--

looks like it started again -_--

it really started again. any fix on this please?

I can confirm that CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) error is happening. Tested from both MacOS and Ubuntu through CLI and jupyter notebook.

It happens on me too.
I got CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 error tested in window and jupyter notebook.

same issue on docker/ubuntu
twitter_parser    | sleeping for 512.0 secs
twitter_parser    | CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

#917

Going to echo what others have said that this probably isn't anything to do with twint.
Not sure if anyone else who uses the Twitter Dev Api received the string of mails recently that myself and a few people I know have?
The initial  one of concern spoke about Dev Apis possibly being abused after being exposed through the browsers cache etc etc.
Seems to me that coupled with the seemingly "new" policies Twitter have been enforcing recently like forcing number verification, banning random accounts and all that I am of the opinion  that they are simply starting to limit the type of full access that we've all become used to & just a general crackdown of sorts
I've been having luck with a solution @yunusemrecatalcam posted in issue #888  https://github.com/twintproject/twint/issues/888#issuecomment-693977671 So big thank  you to him and the Dev of the project mentioned. Works nicely with a rotating pool, speeds are not that bad at all
And of course Big thank you to @pielco11 and everyone involved for your awesome work on Twint. It really is appreciated!

I have also faced the same issue. Twint is not working anymore and giving the error as given in question. How to get rid from this. Is there any alternate ?

looks like it started again -_-

Update for all interested people: Looks like this issue has already been resolved an unknown time ago, if you clone the current version of twint, the noDatExcepting error will not occur any longer. (via https://github.com/twintproject/twint/issues/983#issuecomment-721922571)
🎉

When will these changes be released on pip?

I need to collect tweets, so I used the same code, but I have a problem with implementation .. Please help me because I am a student and the time is very short to complete the project.
import twint
import nest_asyncio
nest_asyncio.apply()
c = twint.Config()
c.Hide_output=True
c.Pandas_clean=True
c.Pandas=True
c.Search="#covid"
c.Since='2020-12-01'
c.Until='2020-12-10'
twint.run.Search(c)

@aliabdmahdi
Instead of installing twint from pypi, use the below command:
pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Uninstalling the Twint tool, then I used this code (pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint).  This error appeared to me. I don’t know the reason ?? .

Uninstalling the Twint tool, then I used this code (pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint). This error appeared to me. I don’t know the reason ?? .

It's because this method first clones the twint from GitHub, that requires you to have git installed on your system.
Instead what you can do is, download the twint repo from GitHub, extract it. And then open the anaconda terminal in the extracted folder.
Then run the following command:
pip install .

It's because this method first clones the twint from GitHub, that requires you to have git installed on your system.
Instead what you can do is, download the twint repo from GitHub, extract it. And then open the anaconda terminal in the extracted folder.
Then run the following command:
pip install .

How open the anaconda terminal in the extracted folder? .

First start the conda terminal, like you previously did. Then change the directory using the cd command.
Google how to change directory in cmd

I put the extracted tool file on my desktop and then went through the installation steps , and there was also an error. What is the problem now?

I put the extracted tool file on my desktop and then went through the installation steps , and there was also an error. What is the problem now?

I put the extracted tool file on my desktop and then went through the installation steps , and there was also an error. What is the problem now?

As per the the error. If you read it it asks you a question.
Do you have got installed?
It's asking because you don't have it installed. Or it's not in your PATH. But I'm gonna guess it's not installed.
Have you installed git?
If not....
https://git-scm.com/downloads

after install git , After performing the steps, this error appeared ..

Don't take this the wrong way but I think you should really consider getting to know the things behind the tools you're trying to use before using them.
For example, in this case, python.
That error indicates that you're probably not using a correct or compatible version of python. Or you've only just done the initial install of conda and not updated it prior to using it.
A quick google search of the error you're getting would help you more than waiting for replies here. And you'll probably learn a a bit along the way which will only be good.
Again, I'm not trying to be mean. But you're going to find yourself running into many issues like this when you start out and people aren't going to want to help you if you don't appear to really try and fix your problem yourself first. Like by just googling your error.
Also because the problem you're having is in no way related to Twint or it's Devs and this space is really for supporting and helping people who experience problems with their tools. It's good practice to respect that.
That error points to countless posts on stackexchange for example. And they always offer up great, detailed information that'll really help you.
I used stackexchange for years before I even posted my first question because almost anything I've ever wanted to ask has already been asked, many times. Lol.
Just some advice. Hope it lands and comes across as intended.

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back.  uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

same here ..

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

I am getting this same error!

I'm afraid, the "CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1" error is back. uninstalling + pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint doesn't work now. Am I the only one for whom this issue is back?

same here, uninstalled it to try this trick and ran into this issue while reinstalling.

UPDATE: for MacOS 11.0 & Python 3.7

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint keeps failing because it can't build the wheel for pycares. If you have homebrew installed, running brew unlink c-ares fixes this issue (according to this SO answer)
after successfully installing twint, I noticed that all of the issues causing this error for my use case (twint.run.Search(query), specifically), were type errors, so I forked the project and fixed them (you can check out the branch on my fork here)

everything works like a charm on my end now!
I'd be more than happy to submit a PR, but decided to hold off until I can get some confirmation that this is a valid solution, or whether other use cases bring up more of these errors.

UPDATE: for MacOS 11.0 & Python 3.7

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint keeps failing because it can't build the wheel for pycares. If you have homebrew installed, running brew unlink c-ares fixes this issue (according to this SO answer)
after successfully installing twint, I noticed that all of the issues causing this error for my use case (twint.run.Search(query), specifically), were type errors, so I forked the project and fixed them (you can check out the branch on my fork here)

everything works like a charm on my end now!
I'd be more than happy to submit a PR, but decided to hold off until I can get some confirmation that this is a valid solution, or whether other use cases bring up more of these errors.

Thanks! I got this error when I try your solution: "Error: No such keg: /usr/local/Cellar/c-ares"

@aurigandrea interesting, what error message do you get when you run the pip install command?

FWIW, we are running twint on an hourly basis (https://github.com/Museum-Barberini-gGmbH/Barberini-Analytics) and haven't seen any major issues during the latest months. We install twint using the pip dependency git+https://github.com/twintproject/twint.git#egg=twint.

Is anyone else still having this issue? I keep getting Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it! on Windows with Python 3.8.8 twint 2.1.20 on every search I run. :(

Is anyone else still having this issue? I keep getting Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it! on Windows with Python 3.8.8 twint 2.1.20 on every search I run. :(

same here, what am I doing wrong?

@charliehawco a fork has been working for me as of this AM -- not sure what's been going on with the core repo though.
Uninstall twint and reinstall with pip install git+https://github.com/woluxwolu/twint.git
See here  for more details.

Hi @eledroos, your fork works brilliantly! only thing is the changes make to the format.py file don't appear to have make to the fork - see issue https://github.com/twintproject/twint/issues/1232


[docs] How to avoid fetching different assets during fluctuations

Consider this scenario:
My phone is connected to WiFi(4g), the power goes off, and my mobile data(2g) kicks into action. If there were multiple assets fetched for 4g, all their low quality equivalents will now be fetched.
This will also happen when I am travelling between places as my data speed keeps fluctuating.
I am not sure what the solution to this problem is but I believe we shouldn't be replacing assets that are already fully loaded.

This is a good gotcha to raise awareness around. We have thought about this and documented one pattern for addressing over-fetching using Service Workers in https://github.com/GoogleChromeLabs/adaptive-loading/tree/master/cra-network-aware-only-if-cached-loading.
I'm going to rename this issue and keep it open as a reminder to add a docs or wiki entry about how to address this particular consideration in a little more detail.

@addyosmani
I think we have one more which is not landed yet.
https://github.com/GoogleChromeLabs/adaptive-loading/pull/27


Appstudio update test-component-pac-gkuo

Pipelines as Code configuration proposal
To start the PipelineRun, add a new comment with content /ok-to-test
For more detailed information about running a PipelineRun, please refer to Pipelines as Code documentation Running the PipelineRun
To customize the proposed PipelineRuns after merge, please refer to Build Pipeline customization

Pipelines as Code CI/test-component-pac-gkuo-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
9 seconds
init

✅ Succeeded
20 seconds
clone-repository

✅ Succeeded
52 seconds
sast-snyk-check

✅ Succeeded
29 seconds
build-container

✅ Succeeded
44 seconds
inspect-image

✅ Succeeded
53 seconds
clamav-scan

✅ Succeeded
46 seconds
deprecated-base-image-check

✅ Succeeded
3 minutes
clair-scan

✅ Succeeded
19 seconds
sbom-json-check

✅ Succeeded
8 seconds
label-check

✅ Succeeded
16 seconds
show-summary

✅ Succeeded
22 seconds
show-sbom

Pipelines as Code CI/test-component-pac-gkuo-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
17 seconds
init

✅ Succeeded
21 seconds
clone-repository

✅ Succeeded
7 seconds
sast-snyk-check

✅ Succeeded
27 seconds
build-container

✅ Succeeded
9 seconds
clair-scan

✅ Succeeded
41 seconds
clamav-scan

✅ Succeeded
9 seconds
sbom-json-check

✅ Succeeded
8 seconds
inspect-image

✅ Succeeded
7 seconds
deprecated-base-image-check

✅ Succeeded
7 seconds
label-check

✅ Succeeded
19 seconds
show-sbom

✅ Succeeded
6 seconds
show-summary


Feature Request: Muting Glassblowing

Glassblowing is a slow progression, often needing like 10k or so per level beyond 75 crafting, so it would be great to mute this loud sound.

Instead of adding a bunch of single item things to mute, use

And add the sound id yourself to the custom mute list at the bottom (not the ambient mute section)

Thank you! If anyone needs this, glassblowing ID is 2724 to mute.


[americastestkitchen] Add support for downloading entire series

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] Searched the bugtracker for similar pull requests
[x] Read adding new extractor tutorial
[x] Read youtube-dl coding conventions and adjusted the code to meet them
[x] Covered the code with tests (note that PRs without tests will be REJECTED)
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.
This PR adds the ability to download entire series of ATK as youtube-dl playlists.

@dirkf could you merge this in for me? I don't have the permissions to do it


nitrpg: misc fixes

Many fixes for NitRPG in prepatation for the next PR wich introduces the use of MongoDB.

+1

but there is errors: http://gresil.org/jenkins/job/CI_github/3646/

:shipit: once tests are fixed

ping about errors

test this please


Git clone a repo and running nbdev_new produces a parsing error in the JSON file

Hey, im nbdev and I'm trying to initialize and use nbdev on our repo. But when trying to initialize it produces a JSON error.
git cloned this repo with SSH https://github.com/Origami-TinyML/tflm_hello_world

pip install nbdev, the latest version
nbdev_new
json parsing error?
all the other commands such as >nbdev_docs will also crash.

But if I download the repo as a zip file and run nbdev it works perfectly fine. Is there something I'm missing out? Thanks in advance!

When running nbdev_new
``error message:
workingdir$  nbdev_new
/home/b/.local/lib/python3.10/site-packages/ghapi/core.py:101: UserWarning: Neither GITHUB_TOKEN nor GITHUB_JWT_TOKEN found: running as unauthenticated
else: warn('Neither GITHUB_TOKEN nor GITHUB_JWT_TOKEN found: running as unauthenticated')
repo = tflm_hello_world # Automatically inferred from git
branch = main # Automatically inferred from git
user = Origami-TinyML # Automatically inferred from git
author = Your Name # Automatically inferred from git
author_email =<EMAIL_ADDRESS># Automatically inferred from git
description = Run "Hello World" of TensorFlow Lite for micro automatically in Docker # Automatically inferred from git
settings.ini created.
Traceback (most recent call last):
File "/home/b/.local/bin/nbdev_new", line 8, in
sys.exit(nbdev_new())
File "/home/b/.local/lib/python3.10/site-packages/fastcore/script.py", line 119, in _f
return tfunc(**merge(args, args_from_prog(func, xtra)))
File "/home/b/.local/lib/python3.10/site-packages/nbdev/cli.py", line 108, in nbdev_new
nbdev_export.wrapped()
File "/home/b/.local/lib/python3.10/site-packages/nbdev/doclinks.py", line 137, in nbdev_export
for f in files: nb_export(f)
File "/home/b/.local/lib/python3.10/site-packages/nbdev/export.py", line 48, in nb_export
nb = NBProcessor(nbname, [exp]+L(procs), debug=debug)
File "/home/b/.local/lib/python3.10/site-packages/nbdev/process.py", line 92, in init
self.nb = read_nb(path) if nb is None else nb
File "/home/b/.local/lib/python3.10/site-packages/execnb/nbio.py", line 57, in read_nb
res = dict2nb(_read_json(path, encoding='utf-8'))
File "/home/b/.local/lib/python3.10/site-packages/execnb/nbio.py", line 16, in _read_json
return loads(Path(self).read_text(encoding=encoding, errors=errors))
File "/opt/conda/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/opt/conda/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/conda/lib/python3.10/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 19 column 13 (char 307)
</details>

Hi,
I had the same problem and fixed it by removing the quotation mark from the repo description, deleting the local files in the repo's root folder, and running nbdev_new again. The error occurs when running nbdev_new and also during workflow actions that fail.


[BUG] gtk::TextView::buffer should return Option<gtk::TextBuffer>.

gtk::TextView::set_buffer takes an Option<gtk::TextBuffer>, but gtk::TextView::buffer returns gtk::TextBuffer.
This is unexpected.
Similar methods (like HeaderBar::set_title_widget) consistently take and return Option<_>

gtk_text_view_get_buffer() is missing the (nullable) annotation in the C code.

Judging from the code this can't actually return NULL:

https://gitlab.gnome.org/GNOME/gtk/-/blob/940248598e5bcef264aff7c18ac5e4bf5ffd2b75/gtk/gtktextview.c#L2289
https://gitlab.gnome.org/GNOME/gtk/-/blob/940248598e5bcef264aff7c18ac5e4bf5ffd2b75/gtk/gtktextview.c#L685

Please re-open with a testcase that fails here if there is one.


Fixed #29899 -- Made autodetector use model states instead of model classes.

Ticket https://code.djangoproject.com/ticket/29899
Hey :wave: ,
I'd suggest reviewing what I did commit-by-commit, in order to be less overwhelmed. The commits probably will need to be squashed before merging, to have one or more meaningful and atomic commits.
The goal is pretty clearly stated through the ticket description.
I started the work off @charettes' https://github.com/django/django/compare/master...charettes:autodetector-model-states, tried to get my head around and work myself through to get it to work.
The things I think need the most review attention and which might be brittle are:

handling swappables
filling the field names of the relations (the logic of choosing the primary key or a {model}_ptr)
basically all the methods added to ProjectState (see the commit named Add and call method of ProjectState to fill relations) to which I'm rather unsure

I basically iterated to what seemed to make sense by making the unit tests pass one-by-one. However, I cannot guarantee that there isn't a regression on an untested corner case :confused:

Let me start by thanking you for reviving this branch and taking the time to adapt it. I left a few comments and questions but over all that looks promising.
I think the commits could be reorganized bit but I think the bulk of the work is there. I like your ProjectState.relation approach. One thing that will need to be considered though is that it should likely be invalidated when reload_model is called even if the latter is not used used the auto-detector. That when both a forward and reverse relation map might come handy.
You might want to peak at recent changes django.db.models.migrations.operations.utils to see if anything there might be useful to move to django.db.models.migrations.utils for reuse.

Thanks a lot for the review! I took the easy changes into account in the last 4 commits (I had to rebase and force-push, sorry :/)
I answered the other comments and hope that you will be able to point me to the right direction.
Thanks again :)

Hi @charettes
I rebased my branch against the latest master and squashed some commits.
I also tried to simplify and remove some logic here and there.
Most notably, I removed the logic filling the FK fields remote name, which was duplicating logic that was inside the rendered FK. However, this meant changing the assertions of some tests. I don't know how critical it is to test it in the autodetector itself, but it seemed not necessarily relevant there.
Looking forward to more reviews :)

After another review I think this is ready to go!
It'd be great to run this branch against large generated project with a few cross referencing models to see how it behaves. After all we were hoping this might help a bit there.
I can't find anything of this kind in https://github.com/django/djangobench/tree/master/djangobench/benchmarks but IIRC @MarkusH had such a script available.

I can only agree with Simon. This is excellent work. Thank you.
As part of issue #24743 I created https://github.com/MarkusH/django-migrations-benchmark .
Unfortunately, I lack the time to have a close look at the changes.
I don't see a full discussion around Simon's comment in https://github.com/django/django/pull/12688#pullrequestreview-400404244 . Is there a chance to move the tracking of which field refers to which other field onto StateApps instead? That would allow for eventually moving the tracking of the reference outside of the migration framework to django.apps.apps.Apps. Which in turn would allow us to use ModelStates for applying migrations. But that's by no means a blocker at this point. If Simon says it's good to go, let's get the chance merged.

Hi @charettes, hi @MarkusH
Thanks a lot for the reviews! :)

I don't see a full discussion around Simon's comment in #12688 (review)

In the PR, I first added a model attribute on the fly to the Field instances, but it was removed and instead, we iterate directly in the only place it's needed (if I recall correctly) here https://github.com/django/django/pull/12688/files#diff-50cc08301fdd233fd64196aa90bc684d96ef557d99af67692995559646de567aR1086-R1093
Did you mean that?

Is there a chance to move the tracking of which field refers to which other field onto StateApps instead?

Sounds like a good idea! However I would prefer doing it in another ticket if possible. The hereby PR is open for quite some time, and I haven't looked at it for a few months. I'd need a bit of time to work on it again, but since I'm working on other projects that wouldn't be soon :)

As part of issue #24743 I created https://github.com/MarkusH/django-migrations-benchmark .
Unfortunately, I lack the time to have a close look at the changes.

I took a quick look at the benchmark and reworked it so that migrate  passes on Django master (see https://github.com/MarkusH/django-migrations-benchmark/compare/master...David-Wobrock:rework-on-latest)
But I'm actually not sure how I would test the non-regression of this patch.
For the migrate speed, I don't see any significant change when trying on master and on this branch (and I didn't really expect it, since only the autodetector is affected by this patch, so makemigrations should only be affected)
Hope that helps and we'll be able to merge this soon 🚀

Hey again @charettes and @felixxm 👋
As you pointed out, the bits of unused code was a little (harmless) mistake.
I rebased and ran again some manual tests of makemigrations to compare the generated migrations between the latest version on PyPI and this branch. I tried to cover as many cases as I could think of: creation, renaming, altering, deletion of concrete models, abstract models, proxy models, managed models, inheritance, FK fields, OneToOneFields with parent link, M2M, M2M with through and through_fields) and couldn't find any difference.
However, I'm still not 100% sure that I didn't miss any case, nor did I bench the performances of detecting a model change on a large project.

@felixxm at this point what is blocking us from moving forward with this PR?

Nothing :+1: with your approval it is at the top of my list. Thanks :star:

@David-Wobrock Many thanks for a great work :rocket:
@charettes Thanks for all reviews and guidance :star:
I pushed small edits and removed unused ProjectState.concrete_apps.

buildbot, test on oracle.


feat: Aria live service to handle SR across browsers

New service to handle the ARIA-LIVE attribute across browsers. For the moment the service and the API are private for internal use only!
Open for discussion and changes.
PR Checklist
Please check if your PR fulfills the following requirements:

[X] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)
[ ] If applicable, have a visual design approval

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] clarity.design website / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: #3822
What is the new behavior?
Creating a service that will create a new DOM element inside the document body. Set the element to have aria-live attribute and project content inside so that the SR will catch the change and announce it.
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information
Close: #3822
Тhe service will create new DOM for every component that requires aria-live support. I manage to test it with about 200 new dom elements and did not find any issues on the performance part. On destroy, for every component, the DOM element must be removed.
Every aria-live container will and must have uniq #id

Adding @coryrylan to do a peer review.

The service works with Safari/VoiceOver and Firefox/NVDA but if used in rapid succession some updates will not be announced. It would be great if the consumer could be responsible for setting up the live region's overall container and a service like this one existed to populate the region. That would reduce our responsibility and empower our consumers to offer users more control of the screen reader user experience via the various options available to the aria-live, aria-atomic and aria-relevant properties


fix: api-server-external-service endpoint bind with svc

What type of PR is this?
/kind bug

What does this PR do?

fix the error of api-server-external-service endpoint bind with svc
Which issue(s) does this PR fix?

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

Need to resolve the error message for CI.


[FNet P0] add PretrainedConfig and unit test

PR types

PR changes

Description

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Swift3 conversion

Converted the code to Swift 3.0, example project runs without any problem.

Hi,
It's  working for me,
verify the travis conf if it's swift 3 ?
thanks for update to 3
see ya

I think you should add @discardableResult to some functions that required it.

@ra1028 I updated the project according to your notes. Thanks for the review.

@eskizyen
I think, there are more functions that which you should to add @discardableResult! 🙇


Animated Jumpscare help!

Describe your problem here.
not exactly a problem with the engine but it was to be asked, how do I make animated jumpscare?
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No response

https://github.com/ShadowMario/FNF-PsychEngine/wiki/Lua-Script-API:-Custom-Sprites-Functions
https://github.com/ShadowMario/FNF-PsychEngine/wiki/Lua-Script-API:-Object-Functions#animations-functions

objectPlayAnimation('funni', 'anim1', false)


Auto complete table name with schema in query editor

SQL Operations Studio Version: 0.26.7

Steps to Reproduce:
Imagine you have the following table. dbo.Customers in your database.
When creating a query in the editor and type the following:
SELECT *
FROM Cust then tab
Autocomplete is curently:
SELECT *
FROM Customers
It would be great if it was:
SELECT *
FROM dbo.Customers
Thanks

Is this still not available? I would love to see this added.


Added a dot clock example script for the voyager

Added a example script which shows how to control the voyagers LEDs.
The example script shows the current time on the voyager in a user defined pattern of colors.

Thank you !


Freezes when Chrome generates userDataDir in workspace

VSCode Version: 1.7 Insider
OS Version: OS X 10.11.6

Steps to Reproduce:

mkdir test && cd test && git init
Run chrome --user-data-dir=.vscode/chrome
VSCode freezes, with git activity bar icon showing count 1. When you restart VSCode it's showing count 99+ for all the files Chrome created. Guess the culprit is git watching folder but too many files are created at the same time.

Found this when I was using "userDataDir": "${workspaceRoot}/.vscode/chrome" in Chrome Debugger.
This doesn't seem to repro if I set userDataDir to "${workspaceRoot}/chrome", though.

I had seen this very rarely while using the Chrome extension and was never able to narrow it down. If you add the chrome folder to .gitignore, does that fix it?

@octref fire up the profiler and find out where time is spend?

@roblourens Adding it to .gitignore doesn't fix it.

@bpasero
I opened the profiler before it freezes, but the profiler ends up freezing too:

Not related to git it seems. Simpler repro:

Run chrome --user-data-dir=.vscode/chrome in your terminal (integrated or external, but pwd must be workspace's root path), for any project.

As the hang is two steps:

My guess is we are watching files in .vscode, but the open Chrome instance messed up our watcher.

.vscode folder is special because we watch it for config changes, this might cause some slow down.

Great catch, we ended up loading >500 JSON files are settings because we look for changes in JSON files recursively below .vscode. I pushed a fix to only support top level JSON files.


Add section on where to land design review docs

Pull Request Template
What are you trying to address
The playbook is missing a section on where design review documentation should land. This addresses issue #409
Description of new changes
Adds section describing where design review documentation should land and why
For all pull requests

[x] Link to the issue you are solving (so it gets closed)
[ ] Label the pull request with the appropriate area(s)
[ ] Assign potential reviewers (you may also want to contact them on Teams to ensure timely reviews)
[ ] Assign the project - Engineering Playbook Backlog
[ ] Assign the pull request to yourself

The issue initially suggests

Tool specific guidance
-- AzDO
-- Github

I'm not exactly sure what this is referring to... any thoughts @wbreza @sbohlen @zmmille2


fix: async fetch and asyncData not trigger ErrorHandler

Codecov Report

Merging #3781 into dev will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #3781   +/-   ##
=======================================
  Coverage   97.81%   97.81%
=======================================
  Files          18       18
  Lines        1145     1145
  Branches      309      309
=======================================
  Hits         1120     1120
  Misses         24       24
  Partials        1        1

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9f3aa2...b28a5c5. Read the comment docs.

@Atinux DOne


⚠ TorrentLeech has degraded performance

In a859558, TorrentLeech ($TL) experienced degraded performance:

HTTP code: 200
Response time: 2639 ms

Resolved: TorrentLeech performance has improved in e760c70 after 11 minutes.


Auto reconnect Feature

For a long-running consumer, auto-reconnects can be very important.
Should be trivial to implement using ZIO's retry capabilities, perhaps a config option to turn off/on auto-reconnect  and allow passing in a Schedule

Sounds excellent, let's do it!

ok great, Ill work on it

@linux-root I should be able to take a look at this within the next week or so

I have been busy since attempting to look into this. However, this is not as simple as it seems, we first need to create a test case to reproduce the issue and then we can look at how to tackle. The underlying Java rabbit library provides reconnects by default so we need to understand what reconnect we are talking about here


lcms port

http://www.littlecms.com/
Little CMS intends to be an OPEN SOURCE small-footprint color management engine, with special focus on accuracy and performance. It uses the International Color Consortium standard (ICC), which is the modern standard when regarding to color management. The ICC specification is widely used and is referred to in many International and other de-facto standards. It was approved as an International Standard, ISO 15076-1, in 2005.

Hi @Agadoul, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.
TTYL, MSBOT;

I checked a bit the library headers https://github.com/mm2/Little-CMS/blob/master/include/lcms2.h#L221 , and it seems that a bit of logic is necessary to correctly export and import symbols depending on the static/shared configuration. How to plan to handle this? For a possible solution, in FreeImage we use a FreeImageConfig.hthat is different between static and shared configuration, see https://github.com/Microsoft/vcpkg/pull/366/files#diff-d0505d14e4a175bfa2cf0aec61f83131R97 .

It looks to me that I just need to add_definitions(-DCMS_DLL_BUILD) in the CMakeLists when dynamic is enabled. Isn't that enough?

I may be missing something, but looking in lcms includes in that case the symbols are correctly exported as CMS_DLL_BUILD is defined, but they will not be imported correctly by third party code because CMS_DLL will not be defined.

Hi, sorry for the long break due to the holiday season. I stayed away from any PC for about 2 weeks. I will resume the PR this week end and have a look at what you mentioned related to the DLL exports.

Long overdue. Here is the updated PR. I used a patch to fix import/export in lcms2.h when dynamic is being build.

No worries ⛷.
Looks great, thanks for the PR!


Adjust txt, csv and json data sources to use the same exec attributes

What was changed

txt, csv and json data sources were adjusted to support both path and glob parameters
the unit tests were updated

Fixes https://github.com/blackstork-io/fabric/issues/176

closing in favor of https://github.com/blackstork-io/fabric/pull/186


RecordArgs and ProductArgs does not support apply method

Upstream bug: https://github.com/scala/bug/issues/10307

  final class PartiallyAppliedNewInstance[Result] extends RecordArgs {
    def applyRecord[Vals](vals: Vals): Result = {
      ???
    }
  }

  /** @usecase def newInstance[Result](vals: Any*): Result */
  def newInstance[Result]: PartiallyAppliedNewInstance[Result] = new PartiallyAppliedNewInstance[Result]

  newInstance[A](x = 1, y = 4.5f, z = "z") // error here

Error:(44, 39) macro method applyDynamicNamed: (method: String)(rec: Any*)Any does not take type parameters.
error after rewriting to newInstance[A].applyDynamicNamed[A]("apply")
possible cause: maybe a wrong Dynamic method signature?
      val a: A = newInstance[A](x = 1, y = 4.5f, z = "z")

Why is this a shapeless issue?

We can create a workaround in RecordArgs by creating an apply method, which should forward to applyRecord.

That's an issue in the Scala compiler


Clang segfaults during build

I'm using xsimd together with the xtensor library. My project builds and runs fine with GCC, however with clang I'm getting compiler segfaults.
This is happening both on linux (Ubuntu clang version 15.0.7) and macOS (Apple clang version 14.0.3 (clang-14<IP_ADDRESS>.1)).
Interesting enough on macOS I get the segfault only on Apple Silicon, not on an Intel machine, although it was segfaulting on the intel machine with the previous version of XCode.
For the moment I can provide this part of the log on linux:
1.	/home/momchil/workspace/audiovention/controlhub/externals/AVxtensor/externals/xsimd/include/xsimd/arch/./xsimd_sse2.hpp:672:77: current parser token ')' 2.	/home/momchil/workspace/audiovention/controlhub/externals/AVxtensor/externals/xsimd/include/xsimd/arch/./xsimd_sse2.hpp:21:1: parsing namespace 'xsimd' 3.	/home/momchil/workspace/audiovention/controlhub/externals/AVxtensor/externals/xsimd/include/xsimd/arch/./xsimd_sse2.hpp:32:5: parsing namespace 'xsimd::kernel' 4.	/home/momchil/workspace/audiovention/controlhub/externals/AVxtensor/externals/xsimd/include/xsimd/arch/./xsimd_sse2.hpp:652:9: parsing function body 'xsimd::kernel::from_mask' 5.	/home/momchil/workspace/audiovention/controlhub/externals/AVxtensor/externals/xsimd/include/xsimd/arch/./xsimd_sse2.hpp:652:9: in compound statement ('{}')
I will try to create a minimum reproducible example. I know this is more of a clang issue than xsimd, but I'd appreciate any possible workaround.
I'm on tag 11.0.0

Hi again, I have a very good update, I've narrowed it down completely - adding the "-Wcast-align" warning causes the segfaults in clang. If anyone arrives here, just disable this warning for a workaround.
You can reproduce the issue by adding
target_compile_options(mandelbrot PUBLIC -Wcast-align)
to the examples CMakeLists and compile the examples.
I will create an issue with clang as well.

I submited a fix upstream :-) https://reviews.llvm.org/D151753


Cache package per executable

I'll probably implement this myself, I'm making an issue so I don't forget. It's probably also a good practice or something. Suggestions on what other features there should be are welcome.
Features:

[ ] Cache the package used per executable (this avoids both the nix-index lookup and the selection menu)
[ ] Add a flag to empty the cache / remove a specific executable

Just as a side note; maybe it helps someone until this is implemented.
I just wrapped a shell script around comma that reads the package with --print-package and then starts a nix shell with that.
This has the advantage, that I don't have to know the binary name (which is not always the same as the package), and I can rerun the binary without any delay, until I exit the nix shell

Just as a side note; maybe it helps someone until this is implemented. I just wrapped a shell script around comma that reads the package with --print-package and then starts a nix shell with that.

https://github.com/nix-community/comma/pull/79


Step numbers are mismatched

Current behavior
The main course page lists Approve pull request as Step 6 (highlighted in screenshot)

Whereas in the course content, Approve the pull request has been erroneously numbered as Step 7. Due to this, the numbering of the steps in the course content are ahead of the main course page by 1, starting with Step 7: Approve the pull request.

Solution
Change step numbers 7, 8, 9, 10, 11, 12 in the course content to step numbers 6, 7, 8, 9, 10, 11 respectively so that they match the main course page.

Thanks for the report, @IshanaRaina!
If you'd like to attempt to fix this, the relevant files are in the responses/ folder, starting with responses/07_approve.md.

Hey @hectorsector, 2 things came up:

The Contributing Guide mentions forking both the course as well as the template repository. But the template repository link on the README.md throws a 404 error. In light of that, I wasn't able to follow the Contributing Guide line by line and instead forked just this repo, updated the step numbers and opened PR #17.

While perusing through the responses 📁, I made yet another observation. When I take the course, the 06_env-variables.md file doesn't show up. The course jumps from Step 5 directly to Step 7 as seen 👇 which is what caused me to open this issue in the first place!

So looking at the flow of steps in the responses 📁, Step 5: Confirm AWS S3 configuration is followed by Step 6: Create and store environment variables which is followed by Step 7: Approve the pull request so maybe in that case we don't want to update the step numbers at all but instead update the code files to trigger the file that was skipped 🤷♀

This was fixed in #17. Thanks @IshanaRaina :octocat:


AzCopy Proxy Issues

I've been having issues setting the proxy info for both AzCopy version 8 and 10. I've tried setting the info based on the instructions documented here and also tried setting them via a AzCopy.exe.config file as documented here. Neither method seems to work.
I'm not getting any errors either. Once I run AzCopy from the command line or invoking it from PowerShell, it just keeps executing forever without providing any kind of meaningful feedback other than "0 bytes transferred'.
Thoughts?

Hi @godlikesme, thanks for reaching out!
Does your proxy require authentication by any chance? Please refer to here for more info.

Is this on Windows?  The next public release is planned to have better proxy detection on Windows.

I also have the very same problem in Windows 10. I tried to make an upload using Azcopy V10 for Powershell, behind a proxy. I found similar links to those that @godlikesme found to set the proxy configuration with no success. Of course, while not connected to the corporate network the upload is working fine.
It surprises me that Azure Storage Explorer with the "AzCopy for Improved Blob Upload and Download" option enabled works fine behind the proxy, after configuring the same proxy settings(Edit -> Configure Proxy).  (Copying AzCopy command from Azure Storage Explorer does not work).

Interesting @ggsggs. Did you, by any chance, follow instructions that tell you to do this:
set https_proxy=<proxy IP>:<proxy port>
Unfortunately, while that's correct in a CMD prompt, "set" in PowerShell doesn't actually set environment variables. It sets a different group of PowerShell variables. (Yes, it confused me too ;-)
For PowerShell, you need to do
$env:https_proxy=<proxy IP>:<proxy port>
@normesta , this sounds like something we should fix in the docs.  Do you agree? (Maybe with two sample commands for Windows. One for CMD.exe and one for PowerShell?)
Finalyl, @ggsggs, as noted earlier in this thread, proxy detection will become much more automated on Windows in the next release.

Awesome. Adding this to the content now. Thx!

Oh, wow. Thank you @JohnRusk!
Your solution worked, (with the quotation marks, perhaps it's self-explanatory for powershell users but it took me three fast tries): $env:https_proxy="<proxy IP>:<proxy port>"
I actually didn't even find the set https_proxy=<proxy IP>:<proxy port> suggestion. It kind of makes me ashamed of my research. I can now see it can be found in Configure, optimize, and troubleshoot AzCopy. :)
Before, I found suggestions regarding creating a AzCopy.exe.config file and adding some specific network configuration lines (this does work for me in App.Config if I use the Azure SDK for C#), same ones as @godlikesme linked in his post.
Thanks guys!

@ggsggs - You didn't miss a thing. That entry didn't appear in the "Configure, optimize, and troubleshoot AzCopy" topic until this morning. I added that entry in response to @JohnRusk 's suggestion - so I'll give a bit thx to @JohnRusk as well!

@normesta - I see!! I feel better now.
Thank you too for your fast replies and updating so promptly the docs :)

Version 10.2.1 is now released with automatic proxy detection on Windows.


ImportError: No module named 'polymorphic.manager'

Hi,
Today i got import error: File "/home/.../shop/models/product.py", line 12, in
from polymorphic.manager import PolymorphicManager
I installed django-polymorphic==2.0.2
It seems this issue related to django-polymorphic's issue. Patch from there also works in my case.

Hi,
Can someone fix this issue?

@dismine
with django-polymorphic==2.0.2 import manager takes a s
 from polymorphic.managers import PolymorphicManager

I know. I mean i just test issue #735 and shop/models/product.py in branch releases/0.12.x still has string from polymorphic.manager import PolymorphicManager. If a maintainer is busy i can create a pull request. I just don't know which branch to use.

django-110 need django-polymorphic==1.3 so in version django-shop 0.12.x  you have to use this for the moment.
does this pose a problem?

We just could add
try:
    from polymorphic.managers import PolymorphicManager
except ImportError:
    from polymorphic.manager import PolymorphicManager

and totally forget about this issue.

does this pose a problem?

Yes, thank you. We can close this issue.

it's planned for the future version:
https://github.com/awesto/django-shop/pull/723/files#diff-794ded6023e23e60359c61837f571483R16


cdh: unhelpful error message on missing socket

If the socket is missing, CDH will print an unhelpful error message instead of what the actual problem is.
# /usr/local/bin/api-server-rest --features all
Starting API server on <IP_ADDRESS>:8006 with features all
Error: Nix error: ENOENT: No such file or directory

Caused by:
    ENOENT: No such file or directory

A user needs to consult strace to find the actual problem:
connect(6, {sa_family=AF_UNIX, sun_path="/run/confidential-containers/cdh.sock"}, 39) = -1 ENOENT (No such file or directory)

Seems that it is caused by ASR? cc @arronwy