Persist UIDs received from Telegram

Short Description:
In order to allow the user to enter username as master in Tegram bot config, store the username/uid relation in datastore - this way, this information surives bot restart.

I found some issues, please do not merge yet.

i think - unneeded duck tape

I think this is a good solution to the issue, although it still requires user to send an initial message to the bot before it will start sending alerts. It's definitely better than what we have now, and probably preferable to just ditching username support entirely.

If we're going to continue to support username as master (and I'm leaning towards thinking we shouldn't), why are we forcing people to put the "@" symbol in there? The Telegram API doesn't use it and all the code needs to strip it out to actually make use of it.
IMHO we should just need "username" (not "@username"), but of course the we'd need to support both for those who already have it set up.
I'm thinking we should just bite the bullet and remove username support altogether. It's flaky even with this modification, and really no less confusing/difficult to set up, given that an initial message is needed to initialize it.

Okay, bugfixing finished, please proceed with review & merge.
As for "why" etc.: current "preferred" method is to enter a numeric user ID. For me, it feels like requiring a user to manually wire the battery and key in the part numebrs into the the car's board computer before starting the motor. Not anywhere near "user-friendly".
I know there is a bot around ("@id_bot") which responds with your chat ID, however it requires user interaction. I have not found any ways yet to find out user id without user interaction - once I find one, I will adjust this logic to not require that "initial message".

@Gobberwart : "@" is not required, the code works with or without it. I strongly disagree on ditching username support, see my previous comment.

Agree it's not user friendly, but it works and the telegram api doesn't provide a simple way to get a user's id from username. Their fault, not ours. I really think we should do everything we can to encourage users to use the id rather than the username until we can find a better way to implement it.
Annoyingly, even if using "username", the user still has to go and find the setting in Telegram to create a username as there isn't one by default.
Happy to approve this in principal, but would prefer if the log messages were not encouraging users to set up in a less-than-ideal way.

Random thoughts re getting ID from username (food for thought):

Telegram api provides no such functionality (according to docs), but...
Telegram app allows searching by username, which must provide the id somewhere, so...
If we can find a way to emulate that search, we should be able to find the id

How to perform a username search without using the app? That's the big question. If we can resolve this, then username should work perfectly.
The @id_bot still requires that you provide the full user contact info (which includes the user id), so that's not going to help. Ditto @get_bot_id.

@Gobberwart - I strongly disagree in one major point with you: the preferred behavior should be for user to provide the user name. It is not the fault of the user that Telegram API does not offer a simple way to convert username to ID - and we should not force the user into retrieving this cryptic ID (not needed for anything else...).
As mentioned above, requiring username vs. user ID is like...requiring the customer to tell the vendor he needs  NaHCO3 - instead of accepting the customer's request for "baking soda". Both terms are valid and in theory, every customer should be able to use Google or Wikipedia or a chemistry book to find the formula, but the common sense says "make sure the cashier knows  NaHCO3 = sodium bicarbonate = baking soda".
Setting the username is done easily in the "Settings" of the Telegram app (https://telegram.org/faq#q-what-are-usernames-how-do-i-get-one), I don't remember the exact place though. It is by all means easier than finding out your user ID.
Yes, for now ( = until we find an automatic way to convert username into ID), a workaround is needed for the first launch, i.e. a message has to be sent from the master. Once we have found a solution to this, the workaround is not needed anymore. As for "@username" vs "username"... Does not make any difference. Per convention, in most places you will see "@username", but technically we can accept both and just strip "@" if it is provided.
Also, two major points are completely being disregarded:

the bot API actually documents "@username" as acceptable for "sendMessage" - it just does not work. Bug in the Bot API, IMHO.
right now, chat_id seems to be equal to user_id (which makes usage of id_bot/get_id_bot possible), but these are different data fields actually. If Telegram decides to disconnect them, our API stops working and will not resume working unti everybody has chnged their config file.

I think we actually agree. Unfortunately the fact is that the Telegram API just doesn't support sending to "username", which is a limitation we have to deal with. Hence why I don't want to encourage people to use it until (if) we get it working the way it should.
Re points 1 & 2...  We'll just have to be creative.
Anyway, I've approved this as is. I'd LIKE to see you remove the username from the log messages (at least until we figure out something better to work around api limitations), but I'm not going to get bent out of shape if you don't.

"Oh, and is there any reason to rediscuss "username vs user id" on every related PR?"
Every? You mean two?

I dunno what's going on here, just opened telegram_handler.py - what a mess :octocat: xD

I think @DBa2016 has addressed all concerns/questions raised, can we merge this?

If nobody has any objections in the next couple of hours, I'll merge and we can deal with the niggles later.

    bot = start_bot(bot, config)
  File "pokecli.py", line 109, in start_bot
    initialize_task(bot, config)
  File "pokecli.py", line 93, in initialize_task
    tree = TreeConfigBuilder(bot, config.raw_tasks).build()
  File "/Users/simba/Projects/pokemon/PokemonGo-Bot/pokemongo_bot/tree_config_builder.py", line 79, in build
    instance = worker(self.bot, task_config)
  File "/Users/simba/Projects/pokemon/PokemonGo-Bot/pokemongo_bot/base_task.py", line 23, in __init__
    self.initialize()
  File "/Users/simba/Projects/pokemon/PokemonGo-Bot/pokemongo_bot/cell_workers/telegram_task.py", line 21, in initialize
    self.bot.event_manager.add_handler(TelegramHandler(self.bot, self.config))
  File "/Users/simba/Projects/pokemon/PokemonGo-Bot/pokemongo_bot/event_handlers/telegram_handler.py", line 182, in __init__
    c.execute("SELECT uid from telegram_uids where username in ('{}', '@{}')".format(srchmaster, srchmaster))

Same here. Fixed it but then encountered repeated "master not numeric" errors, and a bot crash when trying to send /info.  [CP 62] [NCP 0.09] [Potential 0.6] [A/D/S 5/8/14]
[2016-09-06 14:54:31] [MainThread] [PokemonGoBot] [INFO] Telegram master not numeric: @gobberwart
[2016-09-06 14:54:38] [MainThread] [PokemonCatchWorker] [INFO] Nice Curveball throw! Used Pokeball, with chance 78.21 (29 left)
[2016-09-06 14:54:38] [MainThread] [PokemonGoBot] [INFO] Telegram master not numeric: @gobberwart
[2016-09-06 14:54:38] [MainThread] [PokemonGoBot] [INFO] Telegram message from gobberwart (244691262): /info
[2016-09-06 14:54:38] [MainThread] [PokemonGoBot] [WARNING] Telegram message received from correct user, but master is not numeric, updating datastore.
initialising db
[2016-09-06 14:54:38] [MainThread] [PokemonCatchWorker] [INFO] Captured Pidgey! [CP 62] [NCP 0.09] [Potential 0.6] [5/8/14] (124/1000) [+120 exp] [+100 stardust]
[2016-09-06 14:54:38] [MainThread] [PokemonGoBot] [INFO] Telegram bot not running, trying to spin it up
[2016-09-06 14:54:38] [MainThread] [PokemonCatchWorker] [INFO] Captured Pidgey! [CP 62] [NCP 0.09] [Potential 0.6] [5/8/14] (124/1000) [+120 exp] [+100 stardust]
[2016-09-06 14:54:38] [MainThread] [PokemonGoBot] [INFO] Telegram bot not running, trying to spin it up
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO]
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Ran for 0:00:51
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Total XP Earned: 330  Average: 23190.90/h
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Travelled 0.00km
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Visited 0 stops
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Encountered 2 pokemon, 2 caught, 1 released, 0 evolved, 0 never seen before ()
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Threw 3 pokeballs
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Earned 100 Stardust
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Hatched eggs 0
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Next egg hatches in 4.09 km
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO]
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Highest CP Pokemon: Pidgey [CP: 62] [IV: 5/8/14] Potential: 0.6
[2016-09-06 14:54:39] [MainThread] [       cli] [INFO] Most Perfect Pokemon: Pidgey [CP: 62] [IV: 5/8/14] Potential: 0.6
[2016-09-06 14:54:39] [MainThread] [PokemonGoBot] [INFO] Telegram bot not running, trying to spin it up
Traceback (most recent call last):
  File ".\pokecli.py", line 802, in <module>
    main()
  File ".\pokecli.py", line 255, in main
    formatted='Forts cached.',
  File "C:\Users\gobbe\AppData\Local\GitHub\Gobberwart\PokemonGo-Bot\pokemongo_bot\event_manager.py", line 76, in emit
    handler.handle_event(event, sender, level, formatted_msg, data)
  File "C:\Users\gobbe\AppData\Local\GitHub\Gobberwart\PokemonGo-Bot\pokemongo_bot\event_handlers\telegram_handler.py", line 212, in handle_event
    if self.master:
AttributeError: 'TelegramHandler' object has no attribute 'master'
[2016-09-06 14:54:39] [MainThread] [sentry.errors] [ERROR] Sentry responded with an error: 'utf8' codec can't decode byte 0x9c in position 1: invalid start byte (url: https://app.getsentry.com/api/90254/store/)
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\raven\transport\threaded.py", line 174, in send_sync
    super(ThreadedHTTPTransport, self).send(data, headers)
  File "C:\Python27\lib\site-packages\raven\transport\http.py", line 47, in send
    ca_certs=self.ca_certs,
  File "C:\Python27\lib\site-packages\raven\utils\http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python27\lib\site-packages\future\backports\urllib\request.py", line 494, in open
    response = self._open(req, data)
  File "C:\Python27\lib\site-packages\future\backports\urllib\request.py", line 512, in _open
    '_open', req)
  File "C:\Python27\lib\site-packages\future\backports\urllib\request.py", line 466, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\site-packages\raven\utils\http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "C:\Python27\lib\site-packages\future\backports\urllib\request.py", line 1284, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python27\lib\httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "C:\Python27\lib\httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "C:\Python27\lib\httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "C:\Python27\lib\httplib.py", line 895, in _send_output
    msg += message_body
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 1: invalid start byte
[2016-09-06 14:54:39] [MainThread] [sentry.errors.uncaught] [ERROR] [u"AttributeError: 'TelegramHandler' object has no attribute 'master'", u'  File ".\\pokecli.py", line 802, in <module>', u'  File ".\\pokecli.py", line 255, in main', u'  File "C:\\Users\\gobbe\\AppData\\Local\\GitHub\\Gobberwart\\PokemonGo-Bot\\pokemongo_bot\\event_manager.py", line 76, in emit', u'  File "C:\\Users\\gobbe\\AppData\\Local\\GitHub\\Gobberwart\\PokemonGo-Bot\\pokemongo_bot\\event_handlers\\telegram_handler.py", line 212, in handle_event']

Reverted. Needs some more testing/bugfixing, ie. delete tables from db and start bot, run bot with different configs for telegram (id/@username for example).

yes, need revert.

Agreed on revert for now, can you guys provide more information on your crashes? In particular, th exact error message (@solderzzc) and the contents of your TelegramTask?

My config is like this:
  {
    "type": "TelegramTask",
    "config": {
      "enabled": true,
      "master": 246713289,
      "alert_catch": ["all"]
    }
  },

It raised an exception, after I switched the master to string of my username, it raised another exception.

What was the error message in your exception?

      {
        "type": "TelegramTask",
        "config": {
          "enabled": true,
          "master": "@gobberwart",
          "alert_catch": {
            "all": {"operator": "and", "cp": 1300, "iv": 0.95}
          }
        }
      },

Initially had same error as @solderzzc. Fixed that by adding table creation sql to TelegramClass, but then got repeated "Master not numeric" messages in log. As soon as I tried sending /info to the bot, crashed with error shown above.
[2016-09-06 14:54:39] [MainThread] [sentry.errors.uncaught] [ERROR] [u"AttributeError: 'TelegramHandler' object has no attribute 'master'", u'  File ".\\pokecli.py", line 802, in <module>', u'  File ".\\pokecli.py", line 255, in main', u'  File "C:\\Users\\gobbe\\AppData\\Local\\GitHub\\Gobberwart\\PokemonGo-Bot\\pokemongo_bot\\event_manager.py", line 76, in emit', u'  File "C:\\Users\\gobbe\\AppData\\Local\\GitHub\\Gobberwart\\PokemonGo-Bot\\pokemongo_bot\\event_handlers\\telegram_handler.py", line 212, in handle_event']

After I restarted bot, it seemed to be working ok, but still generating a heap of "master not numeric" messages every second or two.

Since much of the code affected was going to be reworked anyway, I collapsed the fix into the PR #5238


requireSpaceBeforeBlockStatements: reworded an error message

Fixes #927

Coverage remained the same when pulling 070efb10e574fb5e240f742816bf18695efb2eae on zxqfox:hotfix/require-space-before-block-statement into b4ab2287c1cd0052bf13a69c61302b8e24933035 on jscs-dev:master.

Closed via https://github.com/jscs-dev/node-jscs/commit/2fca14a5dc2ac1f8db29ed4864684f25bdd0f674


Fix any_image_view::const_t

Description
Fix any_image_view<>::const_t that currently returns an invalid type, see the provided test case that fails on develop.
Tasklist

[x] Add test case(s)
[x] Ensure all CI builds pass
[x] Review and approve

I thought using inheriting constructors was the right think to do, but it looks like it fails with GCC5, is it C++11 feature complete?

https://gcc.gnu.org/projects/cxx-status.html#cxx11 says

GCC 4.8.1 was the first feature-complete implementation of the 2011 C++ standard

However, we have dropped support from GCC 4.8 as its C++11 support is unusable for GIL (and Boost.MP11 too), see https://github.com/boostorg/gil/pull/296
Perhaps GCC 5 is lacking or suffering a bug.

Do you want to support GCC5, in which case I can revert to declaring every constructors (including the missing one to support initializing construction)?

Do you want to support GCC5

Good question.
Dropping support for GCC 5 could be part of switching to C++17 which was seeded on the Slack, https://cpplang.slack.com/archives/CSVT0STV2/p1588870474195500
I'd not object it, but we need to collect opinions from others (may be worth posting to Boost or Boost GIL mailnig list)
/cc @chhenning @lpranam @stefanseefeld

Not sure if this is the one, but there are bugs related to inheriting constructors in GCC 5:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=62310

@sdebionne Any workaround to keep compatible w/ GCC 5 is best option for the time being.

I am OK with both approaches, aka workaround or dropping GCC 5, but the later and making C++ 14 the new language std requirement is more appealing to me.

@sdebionne Yes, let's prepare for the step towards C++14, https://lists.boost.org/boost-gil/2020/10/0469.php

@mloskot To clarify for 1.75, are we dropping GCC 5 (in which case the checks should pass and this PR could be merged) or do you prefer a workaround? Or are we too late anyway?

1.75 has just been frozen for beta.

@sdebionne

To clarify for 1.75, are we dropping GCC 5

No, I don't think it's right to drop GCC 5 or start requiring C++14 from 1.75.
My plan is to add notice to Boost 1.75 release notes that GCC 5 support will be dropped in next release and C++14 will likely be required version too.

I've announced our plans in the release notes, 8b1c2d3ea4eb5bf0a1e7e093862962313dd6c2bb and https://github.com/boostorg/website/pull/562

If I am correct, I need to close and reopen the PR to get the updated pipeline without GCC5? Or is there a better way?

@sdebionne I think you need to rebase your sdebionne:fix-any-image-view-const-t branch and git push --force it to update this PR, then it will rebuild.

Done. But I still see gcc-5 on Azure and gcc-4.9 on Github Actions, both versions probably impacted by the bug mentioned before.

@sdebionne The #572 has been merged. If you could update this PR with the develop, then it should build fine.

How should I interpret the result of the CI:

Azure (ubuntu1604_gcc6_cxx14_cmake) fails
GA (gcc-6, 11,14,1z, ubuntu-16.04, g++-6) passes

That is a puzzle! I can't get why the subimage_view fails there. I'll have a look later tonight.

@mloskot Thanks a lot for looking into this! I forgot that we have two build systems... I'll update the Jamfile.
I am a bit puzzled with the error reported by gcc-6, I'll see if I can find a workaround since I think we need to support that version.

@sdebionne

I forgot that we have two build systems

Yes, it is a bit PITA to maintain the two.
The Jamfile-s for B2 are the 'official' build configuration since it is required for integration with Boost.
The CMake configuration is what we maintain for our own (and contributors) convenience.
My plan is two-fold:

Replace our custom CMakeLists.txt-s with the semi-official CMake for Boost https://github.com/boostorg/cmake, or at least integrate GIL with that new infrastructure.
Further simplification of our CI: ideally, if we have single one to maintain, that is, GitHub Actions only, but it needs to cover a) all tests (core and extensions, including IO); b) all compilers we want; c) B2 and CMake

I'll see if I can find a workaround since I think we need to support that version.

Well, it would simplify things if we can support GCC 6,
Alternative, is to bump the C++ requirement to C++17, instead of C++14 as we have it now.
I, personally, am usually more eager to move from C++11 to C++17 than from C++11 to C++14.
So, I'd not mind, especially that next GIL release will happen around Boost 1.77 or 1.78, so plenty of time.
/cc @lpranam @simmplecoder

Hi @pdimov, I am encountering an issue which I think is related to inheriting constructors from Variant2 variant and older compilers (clang < 4 and gcc < 7). Any chance that this rings a bell?
The pattern is as follow:
template <typename ...Views>
class any_image_view : public variant2::variant<Views...>
{
    using parent_t = variant2::variant<Views...>;

public:
    using parent_t::parent_t;
    ...
}

Here is the error message simplified for readability:
./boost/gil/image_view.hpp:100:81: error: no member named 'pixels' in 'boost::gil::any_image_view<...>'
    image_view(View const& view) : _dimensions(view.dimensions()), _pixels(view.pixels()) {}
                                                                           ~~~~ ^
./boost/variant2/variant.hpp:640:90: note: in instantiation of function template specialization 'boost::gil::image_view<...>::image_view<boost::gil::any_image_view<...>' requested here
    template<class... A> constexpr variant_storage_impl( mp11::mp_size_t<0>, A&&... a ): first_( std::forward<A>(a)... )
                                                                                ^
./boost/mp11/algorithm.hpp:1039:28: note: in instantiation of function template specialization 'test_any_image_view_init_ctor::operator()<boost::gil::image<boost::gil::pixel<unsigned char, boost::gil::layout<boost::mp11::mp_list<boost::gil::gray_color_t>, boost::mp11::mp_list<std::integral_constant<int, 0> > > >, false, std::allocator<unsigned char> > >' requested here
    return (void)A{ ((void)f(T()), 0)... }, std::forward<F>(f);

I guess I need to extract a separate test case on Godbolt to get some help...

@pdimov Here is a godbolt that, I think, shows our problem with (clang < 4 and gcc < 7 and probably MSVC).
https://godbolt.org/z/fs44cz
I am not saying that it's related to Boost.Variant2 but would really appreciate your insight.

It's indeed related to Variant2; please open an issue there. I don't know yet whether it can be fixed, but I'll give it a try. The problem is that the variant constructor performs overload resolution over all alternatives and this instantiates the greedy foo(T const&) converting constructor.

@mloskot I am glad that this issue is now fixed. Maybe deprecating the old compilers was not necessary after all? Side question: isn't the new codecov check a bit picky? If we want to save polar :bear:, we might not want to recompile for 10 platforms/compilers for a -0.01% regression in code coverage...

Maybe deprecating the old compilers was not necessary after all?

You mean GCC 5 and switch to C++14?
Well, let's agree that we're slowly heading towards the convenience of C++17 :-)

Side question: isn't the new codecov check a bit picky?  We might not want to recompile for 10 platforms/compilers for a -0.01% regression in code coverage...

Hmm, I'm not sure what you mean, since
https://github.com/boostorg/gil/blob/a82af6d25de304bd7482fdfece3b6f775f2832b9/.github/workflows/coverage.yml#L41
but we definitely don't want to generate coverage building with N compilers, just one should be enough.
@avinal Perhaps you, as contributor of the Codecov GA, could comment on that?

@avinal Perhaps you, as a contributor of the Codecov GA, could comment on that?

Hello @mloskot and @sdebionne, I don't think code coverage is building on more than one platform. There is a single configuration and only one build job.
https://github.com/boostorg/gil/blob/a82af6d25de304bd7482fdfece3b6f775f2832b9/.github/workflows/coverage.yml#L46-L49
For further clarification, you can compare it with CI config, there are N configs in the matrix and it runs for every single of them.
https://github.com/boostorg/gil/blob/6da59cc3351e5657275d3a536e0b6e7a1b6ac738/.github/workflows/ci.yml#L127-L131
Also, CI and Code Coverage jobs are in a different workflow. If you don't want to recompile for all, you can force stop other jobs, in that case, only one compilation job will run, and only that job is needed for code coverage.
Now when I see the config, I think, this line is kinda unnecessary, we can remove it if you think it is of no use in the compilation
https://github.com/boostorg/gil/blob/a82af6d25de304bd7482fdfece3b6f775f2832b9/.github/workflows/coverage.yml#L41

Well, let's agree that we're slowly heading towards the convenience of C++17 :-)

Absolutely!

Side question: isn't the new codecov check a bit picky?  We might not want to recompile for 10 platforms/compilers for a -0.01% regression in code coverage...

Hmm, I'm not sure what you mean, since

Sorry I wasn't clear, I meant that this PR fails for a -0.01% in code coverage. I could fix this eventually, but then we would need to rebuild for n platforms. Or is Codecov just an indicator that should not block a PR to be merged?

@sdebionne

Or is Codecov just an indicator that should not block a PR to be merged?

A very good point!
Yes, it should be considered as an indicator only with informational purpose and it should not block the merge.
@avinal Thank you for the explanation. By the way, do you know if we can configure the coverage threshold e.g. set it to 1-2 %? Alternatively, we could allow the Codecov built pass regardless of the result.

Welcome @mloskot, I think this https://docs.codecov.io/docs/codecovyml-reference#coverageprecision is what you are talking about, not sure though.

LGTM. Shall we merge?

If you are asking me, I would say yes! Or do I need to rebase?

Thanks

@sdebionne

I am OK with both approaches, aka workaround or dropping GCC 5, but the later and making C++ 14 the new language std requirement is more appealing to me.

Let's do it for Boost 1.80. I have moved the planning towards C++14/17 to discussion here https://github.com/boostorg/gil/discussions/676


chore(deps): upgrade dev dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-dev-dependencies-1.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 3092

Questions ?
Please refer to the Backport tool documentation


style: replace all vars with const or let

Coverage remained the same at 33.816% when pulling bea509e6152cf9c853ed0fa4428fdc403c8d8031 on style into 31b556a832e42f71d7e9b8430c50cd17b8b7f927 on main.

Coverage remained the same at 33.816% when pulling bea509e6152cf9c853ed0fa4428fdc403c8d8031 on style into 31b556a832e42f71d7e9b8430c50cd17b8b7f927 on main.


Set legend for predicted and actual value when visualizing the meta-regression model

We don't have a clear indication of predicted vs actual values when plotting the data; A little legend would be nice.


[Don't Merge] Add bulk import/export APIs

Hi @andrerod, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (andrerod). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@stankovski Updated. Could you have a look ?
Thanks


Problem when adding account

There's currently a problem when adding an account, i'm going to fix this in the next update by updating the error handling for it

should be fixed in v.2.0.0


fixed Responsive Error in login page

I have solved the issue #3250
Before :

After :

@fredvd @stevepiercy

@SayanDeveloper you need to sign the Plone Contributor Agreement to merge this pull request.
Learn about the Plone Contributor Agreement: https://plone.org/foundation/contributors-agreement


update docs to explain configuring vite hmr

Environment
------------------------------
- Operating System: Linux
- Node Version:     v18.19.0
- Nuxt Version:     3.7.4
- CLI Version:      3.10.0
- Nitro Version:    2.6.3
- Package Manager<EMAIL_ADDRESS>- Builder:          -
- User Config:      vite, ssr
- Runtime Modules:  -
- Build Modules:    -
------------------------------

Reproduction
Reproduction github PR https://github.com/hutchic-org/nuxt-37-ws/pull/1
git clone git@github.com:hutchic-org/nuxt-37-ws.git
cd nuxt-37-ws
make run
ctrl^c
make exec
netstat -tupln

expected result
netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:3000            <IP_ADDRESS>:*               LISTEN      19/node
tcp        0      0 <IP_ADDRESS>:39775         <IP_ADDRESS>:*               LISTEN      30/node
tcp6       0      0 :::22301                :::*                    LISTEN      30/node

actual result
netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:39931         <IP_ADDRESS>:*               LISTEN      30/node
tcp        0      0 <IP_ADDRESS>:3000            <IP_ADDRESS>:*               LISTEN      19/node

Describe the bug
The following nuxt.config.ts on 3.7.3 gets a node process active on the port specific by port but that's no the case as of 3.7.4
// https://nuxt.com/docs/api/configuration/nuxt-config
export default defineNuxtConfig({
  vite: {
    server: {
      hmr: {
        protocol: 'ws',
        port: 22301,
        clientPort: 22301,
        path: 'hmr/',
        timeout: 3,
      }
    }
  },
  ssr: false,
})

Additional context
No response
Logs
No response

Possibly related to this Vite issue? https://github.com/vitejs/vite/issues/3737
Apparently middlewareMode: 'ssr' is deprecated and should just be middlewareMode: true

@Soviut still no joy though that issue looks very closely related

I ran a git bisect reproducing the issue in the playground and it says https://github.com/nuxt/nuxt/commit/f842da5b905b2a4f0dd28dfa9dfbb51fbb0de3de is the commit at fault

Might be related: https://github.com/nuxt/cli/pull/189

@manniL that looks to have done it.
can / should we ninja edit the relevant docs / CHANGELOG?

FYI was just bitten by this as well in my org, running Nuxt 3.11.2
Even when using the default Vite port I was getting weird errors in the browser as the host was being overridden to localhost and port was undefined
The env variable NUXI_DISABLE_VITE_HMR that manniL shared that disables the override behavior did it for me too.
I think we should more explicitly document this behavior as it was pretty obscure to me that the expected default Vite HMR wasn't respected by Nuxt 3 and when trying to configure it myself it was still being overriden by a Nuxt hook, requiring knowledge of this NUXI_DISABLE_VITE_HMR env var to fix it.
Are we waiting for anything else to have this documented in the official docs?

Any time estimation on when this can be achieved. I've been trying to make HMR work for some weeks and have no clue.
I already set the NUXI_DISABLE_VITE_HMR and if I don't set anything, I receive a  ERROR  WebSocket server error: Port is already in use in the dev console. If I try to set a port on vite.server.hmr.port the error goes away but still no HMR


CI linting

Method:

Run global isort auto-fix
Run global remove-unused-imports
Run flake8 and fix the obvious things
Ignore the sql folder (this should be deleted)
Ignore docstrings for now
Noqa rather than refactor
Create GitHub CI with isort, black, and basic flake8 test

Closing due to excessive merge conflicts, may be repeated/reapplied in future.
cc @WillDaSilva

Superseded by https://github.com/meltano/meltano/pull/6481


FIX: tracer config parameter caused setup error (circular JSON).

Should be PR-ed through RC branch to be correctly tested


Error: "Double requirement given" when using requirements.txt for assignment1

Following requirements are mentioned twice in reuirements.txt that comes with the zip file for assignemnt 1.

jupyter (1.0.0 both times)
pillow (3.1.0 and 3.0.0)

This gives the error "Double requirement given" when running pip install -r requirements.txt

I see the same problem.  Removing the Pillow==3.0.0 line and one jupyter==1.0.0 line worked.
If someone knows a way to get pip to do a "dry run" (find out what it would do, without actually e.g. downgrading matplotlib from 1.5.1 to 1.5.0), let me know....


Ember Client sometimes failing with org.http4s.ember.core.EmptyStreamError: Cannot Parse Empty Stream

I discussed this on Discord with @RaasAhsan here a while back: https://discord.com/channels/632277896739946517/632286375311573032/844933274890534974
This same error was raised in another service with a different 3rd party client, using http4s-ember-client 1.0.0-M23. I've decided to revert back to using Blaze in this case for the time-being, but wanted to raise the issue here.
It was suggested in the discord thread that this wasn't occurring in other client implementations due to some baked-in retry behaviour that ember is perhaps lacking

I'm experiencing this as well.

This issue caused our app to throw an OOME. Fixed it by switching to Blaze backend.

Thanks, all.  Does anybody have a reproduction?  Or if it's an OOME, any clues from a heap dump?

When I was experiencing it, it was only cropping up when calling 3rd party APIs - so it was hard to reproduce by any means other than firing a load of requests at the API that I could see was causing it. I appreciate that is not very helpful for debugging the issue though 😬

@rossabaker The retained heap was cumulated in the buffeRef of fs2.io.tcp.SocketGroup under the EmberConnection

Regarding how this error could impact the retained heap, there are 2 areas that concern me a bit:

No action in case of failure on Exitcase Failure in here
Under the cleanup method, there's a release step that comes from the ClientHelpers.requestKeyToSocketWithKey where in case of a requestKey with http scheme a .pure on Resource is called (here). This seems to result in a Resource which "has a no-op release."

Please let me know if I got anything wrong or have an idea where things could go sideways.

I can't add much to the discussion, except I see this error as well with http4s v0.23.1 and the ember client implementation. The HTTP client is fetching data from one of our own services. The service is implemented with http4s 0.21.22, and it is using the blaze server implementation. The service is running behind an Apache reverse proxy. The error occurs approximately with every second request. I don't see any errors if I use the blaze client implementation.

The service is running behind an Apache reverse proxy.

By chance do you know if it is using HTTP/1.0 ?

The service is running behind an Apache reverse proxy.

By chance do you know if it is using HTTP/1.0 ?

The response status line states it is using HTTP/1.1

@usommerl  Trying to get a local setup going to see if I can reproduce because your case sounds fairly promising. Can you provide any more detail about your set up? What version fo apache proxy, any detail around the configuration?  If you could provide sample HTTP request/response logs as well that would be great

If anyone can capture and share a TCP/wireshark dump of this interaction, that would be immensely helpful as well. I've tried to repro with some different request/response patterns but haven't gotten a hit yet

@RaasAhsan I was able to reproduce the issue with a static text file that is served by Apache. So it has nothing to do with the service behind the proxy. I've only seen the error with GET requests so far, but I haven't tried other HTTP methods yet.
I have created a small example application that is repeatedly querying the Apache service that is running on one of our test servers. Maybe you can use it to reproduce the error on your machine. You may need to play around with the frequency of the requests. On my laptop, I don't see the error if I query the Apache every second. However, If I decrease the frequency, it fails after a couple of requests.
I'm glad if I can provide more information, but I will be a little slow to respond in the next two weeks because I'm on vacation and probably offline most of the day.

@usommerl I'm seeing the errors! It seems like you may be running a server somewhere for this, so I'll try to sort this out ASAP :) Many thanks for getting this set up!

@RaasAhsan You don't need to hurry. The server will be running anyway and most of our team is on vacation, so the server setup will probably not change within the next one or two weeks.

It seems that this bug is an instance of a larger issue that we've been tracking for awhile, namely, request retries when the peer has closed the connection after we write a request (https://github.com/http4s/http4s/issues/4798). I've been spending some understanding how other HTTP client libraries behave in this situation, and it seems most of them at the very least will retry idempotent requests (GET, DELETE, HEAD, PUT requests). I'm going to try out a few more libraries outside the Java ecosystem but I think we're slowly converging to a solution here. thanks again for the server!

So here I've written something we can either incorporate into the client or operate differently. As of now we have the commented available so this is avaiable for user code as well. I see no other fix other than something like this.
https://gist.github.com/ChristopherDavenport/c089f842952aac10fd5f3f555e43b499

To sound smarter than I am, I like to quote standards.  RFC2616, which is obsolete, said:

If an HTTP/1.1 client sends a request which includes a request body, but which does not include an Expect request-header field with the "100-continue" expectation, and if the client is not directly connected to an HTTP/1.1 origin server, and if the client sees the connection close before receiving any status from the server, the client SHOULD retry the request.

Cool.  We have permission to retry POST /gimme/some/money.  But along comes RFC7230, the current law of the land:

A user agent MUST NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied

Drat.  It turns out if we get paid twice, we can get sued.  But, gee, we sure want to be paid once.
This abandoned retry spec is interesting, particularly its coverage of what various clients do.  Firefox is timid and Chromium and Curl boldly retry.  So much for appealing to experts.
There is also the draft Idempotency key specification, which is clever.  And our Retry middleware honors it.  The problem is the server has to as well.
Finally, there is the HTTP semantics draft, which updates the RFC7230 that makes us sad.  Does it have words of comfort?

A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.
...
Some clients take a riskier approach and attempt to guess when an automatic retry is possible. For example, a client might automatically retry a POST request if the underlying transport connection closed before any part of a response is received, particularly if an idle persistent connection was used.

💩

As for @ChristopherDavenport's gist, I think it roughly implements what the latest draft calls "the riskier approach", like Chromium and Curl and unlike Firefox.  My concern with it is whether EndOfStreamError can be thrown after bytes have been read from the response.  If the Content-Length is longer than the body in the current Ember parser, what happens?  I think we could only want an automated retry if this error happens immediately.
Given the ambiguity of the specification and the fact we can go wrong either way, we should probably decide if we're Team Firefox or Team Chromium with our semantics, provide an easy way to flip the switch, and describe how and why in the docs.

I think the implementation Chris has right now only retries requests we know are idempotent (by method or presence of idempotency key), so a POST request that didn't have a key would never be retried.

My concern with it is whether EndOfStreamError can be thrown after bytes have been read from the response. If the Content-Length is longer than the body in the current Ember parser, what happens?

EndOfStreamError is only raised when the peer closes the connection on us and no bytes have been sent yet. If we read some bytes and then observe the close, it raises a ReachedEndOfStream error. So we have the means to implement the risky strategy if we'd really like.
The spec also says:

A client SHOULD NOT automatically retry a failed automatic retry.

This is a little vague, but do we interpret this to mean that a retry should only be performed at most once?

Given the ambiguity of the specification and the fact we can go wrong either way, we should probably decide if we're Team Firefox or Team Chromium with our semantics

👍

I think the implementation Chris has right now only retries requests we know are idempotent (by method or presence of idempotency key), so a POST request that didn't have a key would never be retried.

Correct.  This is the Firefox approach.  We could create a named strategy that makes it easy to switch to the riskier Chromium/Curl approach, but I think where there is ambiguity, we should err on the side of the more conservative default.

EndOfStreamError is only raised when the peer closes the connection on us and no bytes have been sent yet. If we read some bytes and then observe the close, it raises a ReachedEndOfStream error.

Yep.  I was conflating ReachedEndOfStream with EmptyStreamError.

A client SHOULD NOT automatically retry a failed automatic retry.

This is a little vague, but do we interpret this to mean that a retry should only be performed at most once?

Yes.  I'm interpreting that rule as binding for our default behavior, and is implemented by @ChristopherDavenport.  Since it's a SHOULD NOT, and since people might get fancy with exponential backoffs and what not, I think users should continue to feel free to do as they please with the Retry middleware.

Closed by #5064

I am still getting this with 0.23.3.  To make it easier to test, I set my idle connection timeout to 5 seconds.  Sure enough first request succeeds, wait 6 seconds, second request always fails.  Then third request succeeds.
server[ERROR] org.http4s.ember.client.internal.ClientHelpers$$anonfun$request$6$$anon$1: Remote Disconnect: Received zero bytes after sending request
server[ERROR] 	at org.http4s.ember.client.internal.ClientHelpers$$anonfun$request$6.applyOrElse(ClientHelpers.scala:134)
server[ERROR] 	at org.http4s.ember.client.internal.ClientHelpers$$anonfun$request$6.applyOrElse(ClientHelpers.scala:133)
server[ERROR] 	at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:263)
server[ERROR] 	at cats.ApplicativeError.$anonfun$recoverWith$1(ApplicativeError.scala:106)
server[ERROR] 	at cats.data.KleisliApplicativeError.$anonfun$handleErrorWith$2(Kleisli.scala:621)
server[ERROR] 	at main$ @ com.comcast.palermo.server.app.Main$.main(Main.scala:55)
server[ERROR] 	at get @ fs2.internal.InterruptContext.$anonfun$eval$1(InterruptContext.scala:91)
server[ERROR] 	at get @ fs2.internal.InterruptContext.$anonfun$eval$1(InterruptContext.scala:91)
server[ERROR] Caused by: org.http4s.ember.core.EmberException$EmptyStream: Cannot Parse Empty Stream
server[ERROR] 	at org.http4s.ember.core.Parser$MessageP$.$anonfun$parseMessage$1(Parser.scala:48)
server[ERROR] 	at cats.data.Kleisli.$anonfun$flatMap$2(Kleisli.scala:52)

Using 0.23.3 http4s, cats-effect 3.2.9, fs2 3.1.2

Not exactly sure what this fix was supposed to do, but experiencing this on a GET request.
  private[ember] object RetryLogic {
    private val retryNow = 0.seconds.some
    def retryUntilFresh[F[_]]: RetryPolicy[F] = { (req, result, retries) =>
      if (emberDeadFromPoolPolicy(req, result) && retries <= 2) retryNow
      else None
    }

    def emberDeadFromPoolPolicy[F[_]](
        req: Request[F],
        result: Either[Throwable, Response[F]]): Boolean =
      (req.method.isIdempotent || req.headers.get[`Idempotency-Key`].isDefined) &&
        isEmptyStreamError(result)

    def isEmptyStreamError[F[_]](result: Either[Throwable, Response[F]]): Boolean =
      result match {
        case Right(_) => false
        case Left(EmberException.EmptyStream()) => true
        case _ => false
      }
  }

isEmptyStreamError is never true in this case I think because of the adaptError further up in ClientHelpers:
    for {
      processedReq <- preprocessRequest(request, userAgent)
      res <- writeRead(processedReq)
    } yield res
  }.adaptError { case e: EmberException.EmptyStream =>
    new ClosedChannelException() {
      initCause(e)

      override def getMessage(): String =
        "Remote Disconnect: Received zero bytes after sending request"
    }
  }

It's no longer an EmptyStreamError but a ClosedChannelException.  Is that expected?

If I make that fix locally (by duplicating retry policy locally) retries seem to work fine.

@matthughes would you like to open a PR to make that fix ? Bonus points for a test :) If not, one of us will try to get something in soon since Ross is running a release train

Can certainly PR the fix; might need some pointers regarding the test.

Thanks! I don't think we have a good place to write this test with all the scaffolding set up today, but we would want to set up a scenario where a server (could be ember-server) has an idle connection timeout that is less than the client's, and make a couple of requests accordingly
The ember server test suites have a similar set up, so maybe you can reuse some of it in ember-client tests: https://github.com/http4s/http4s/blob/23bd54876aa439bfc4196fbb131fb63fc2778c55/ember-server/shared/src/test/scala/org/http4s/ember/server/EmberServerSuite.scala
Again, feel free to do as much or as little as your free time permits :)

Wrote this test but could use a little help.  I had to put it in EmberServerSuite because ember-client doesn't have access to ember-server.  Unless there is some other server backend I could use trivially:
At least how the bug is reproduced for me, every request made after the client closes a connection due to idle timeout would fail with empty stream.  So I set a low idle timeout (1 second), make a request, sleep 2 seconds, and then make subsequent request.  However, test is failing while making the second request with:
 java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
    at sun.nio.ch.IOUtil.write(IOUtil.java:79)
    at sun.nio.ch.IOUtil.write(IOUtil.java:50)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishWrite(UnixAsynchronousSocketChannelImpl.java:591)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:195)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
    at sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:312)
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:834)
    at async @ fs2.io.package$.asyncYield(io.scala:246)
    at guarantee @ fs2.internal.CompileScope.$anonfun$interruptWhen$1(CompileScope.scala:384)
    at flatMap @ fs2.io.tcp.SocketGroup.go$3(SocketGroup.scala:334)
    at apply @ org.http4s.ember.server.internal.StreamForking$.$anonfun$forking$1(StreamForking.scala:44)
    at unsafeToFuture @ munit.CatsEffectSuite$$anonfun$1.applyOrElse(CatsEffectSuite.scala:41)
    at acquire @ org.http4s.ember.server.internal.StreamForking$.runInner$1(StreamForking.scala:78)
    at apply @ org.http4s.ember.server.internal.StreamForking$.$anonfun$forking$1(StreamForking.scala:44)
    at withPermit @ fs2.io.tcp.SocketGroup.fs2$io$tcp$SocketGroup$$read0$1(SocketGroup.scala:290)
    at map @ fs2.internal.CompileScope.interruptibleEval(CompileScope.scala:413)
    at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:503)
    at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
    at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
    at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)
    at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:436)

  fixture().test("empty stream") {
    case (server, _) =>
      import org.http4s.dsl.io._
      import org.http4s.client.dsl.io._
      import scala.concurrent.duration._

    val body: Stream[IO, Byte] =
      Stream.emits(Seq("hello")).through(fs2.text.utf8Encode).covary[IO]

    val uri = Uri
      .fromString(s"http://${server.address.getHostName}:${server.address.getPort}/echo")
      .toOption
      .get
    val request = GET(body, uri)
    EmberClientBuilder.default[IO].withIdleConnectionTime(1.seconds).build.use { client =>
      for {
        r1 <- client.fetchAs[String](request)
        _ <- IO.delay(println("sleeping"))
        _ <- IO.sleep(2.seconds)
        _ <- IO.delay(println("fetching again"))
        r2 <- client.fetchAs[String](request)
        _ <- IO.delay(println("fetched"))
      } yield assertEquals("hello", r1) && assertEquals("hello", r2)
    }
  }

@matthughes I've seen Broken pipe errors occur when trying to write to a socket that has already been closed. I've seen this in other tests as well, which may be a sign of some deeper bug. Could you open a PR and we can investigate further there ?

Here's the PR: https://github.com/http4s/http4s/pull/5247

Hi, is this still an issue or has it been fixed by the PRs mentioned above?

@channingwalton I'm sorry to hear that. Can you share a reproducer by any chance? That would be a big step towards a solution, thanks!

@channingwalton thanks, what version of http4s-ember-client is this btw?
It is definitely an ember error, this one specifically:
https://github.com/http4s/http4s/blob/4f4cbaa7a8a7ea5d84fdbbcd09306522b6cfdb7d/ember-client/shared/src/main/scala/org/http4s/ember/client/internal/ClientHelpers.scala#L174-L180

what version of http4s-ember-client is this btw

0.23.16

Just for my clarification, have all the issues described above (https://github.com/http4s/http4s/issues/4935#issuecomment-923978913) been resolved and we are hunting a new problem, or is it the same problem?
Also, when the client reports Remote Disconnect: Received zero bytes after sending request, has the remote really disconnected?

We have two services which started having these errors after upgrading to 0.23.16. They did not have the errors prior to the upgrade.
Could there have been a regression?

Just for my clarification, have all the issues described above (#4935 (comment)) been resolved and we are hunting a new problem, or is it the same problem?

I don't know. https://github.com/http4s/http4s/pull/5247 purportedly solved it, but there is still the occasional report.

Also, when the client reports Remote Disconnect: Received zero bytes after sending request, has the remote really disconnected?

If the remote is sending end-of-stream then it has effectively disconnected.

We have two services which started having these errors after upgrading to 0.23.16. They did not have the errors prior to the upgrade.
Could there have been a regression?

Interesting, this is really helpful. @channingwalton which version did you upgrade from? The 0.23.16 release had no significant changes besides adding Scala Native support.

which version did you upgrade from?

0.23.15

Are you able to reproduce with http4s 0.23.15 with FS2 3.3.0?

I can try that.
I am trying to reproduce it but not succeeding with a small test from our service. Maybe it happens under load? Maybe it's the 3-rd party we are querying failing under load combined with something about EmberClient.
I want to try to downgrade to 0.23.15 to see if the problem goes away to confirm it's something between 0.23.15 and 0.23.16.
I'll be back :)

@armanbilge Not sure if related to this, but I am seeing something similar with ember client here.

@ahjohannessen thanks for linking, do you have a repro for that?

@armanbilge Yep, https://github.com/daddykotex/smithy4s-client-middleware/compare/main...ahjohannessen:smithy4s-client-middleware:wip-repo-empty-body-json

I pushed some changes here: https://github.com/daddykotex/smithy4s-client-middleware/tree/wip-repo-empty-body-json
I can't share much now, but will come back later (kids are after me!)

Basically, we use smithy4s to generate a server (on top of a http4s) and a client (http4s also). There are two routes but I've tried different encoding of the operation (no path param, no query param, required body) and the result was always the same.
I've added different variant (functions that can be called to reproduce) that shows:

hello then hello2 fails, as well as calling hello twice
calling hello2 before hello, works
calling hello2 twice works, but is very slow

Thank you both, this is very helpful. I switched your reproducer to use the Scala.js version and I was able to replicate the error reported in this issue. So it is a good lead!
https://github.com/armanbilge/smithy4s-client-middleware/tree/wip-repo-empty-body-json-sjs

sbt:MyApi> run
[info] Running com.example.Main.
HelloOutput(Hello call 1!)
org.http4s.ember.client.internal.ClientHelpers$$anonfun$request$7$$anon$1: Remote Disconnect: Received zero bytes after sending request
  at org.http4s.ember.client.internal.ClientHelpers$$anonfun$request$7.applyOrElse(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:164096:12)
  at org.http4s.ember.client.internal.ClientHelpers$$anonfun$request$7.applyOrElse(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:164108:15)
  at scala.PartialFunction$AndThen.applyOrElse(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:142477:48)
  at {anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:223763:85)
  at delay @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:73903)
  at flatMap @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:73911)
  at async @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:188905)
  at defer @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:188901)
  at defer @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:111177)
  at main @ com.example.Main.main(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:10258)
  at productR @ com.example.Main$.helloTwice(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:113187)
  at delay @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:188199)
  at flatMap @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:73872)
  at run @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:75550)
  at flatMap @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:231775)
  at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:96305)
  at timeoutTo @ org.http4s.ember.core.Util$.timeoutToMaybe(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:96313)
  at getAndSet @ org.typelevel.keypool.KeyPool$.destroy(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:49683)
Caused by: org.http4s.ember.core.EmberException$EmptyStream: Cannot Parse Empty Stream
  at {anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:34962:41)
  at map @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:231647)
  at delay @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:73903)
  at flatMap @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:73911)
  at async @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:188905)
  at defer @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:188901)
  at defer @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:111177)
  at main @ com.example.Main.main(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:10258)
  at productR @ com.example.Main$.helloTwice(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:113187)
  at delay @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:188199)
  at flatMap @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:73872)
  at run @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:75550)
  at flatMap @ <jscode>.{anonymous}()(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:231775)
  at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:96305)
  at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(/workspace/smithy4s-client-middleware/target/scala-2.13/myapi-fastopt/main.js:96305)
[error] org.scalajs.jsenv.ExternalJSRun$NonZeroExitException: exited with code 1
[error]         at org.scalajs.jsenv.ExternalJSRun$$anon$1.run(ExternalJSRun.scala:200)
[error] stack trace is suppressed; run last Compile / run for the full output
[error] (Compile / run) org.scalajs.jsenv.ExternalJSRun$NonZeroExitException: exited with code 1
[error] Total time: 2 s, completed Oct 14, 2022, 9:45:57 PM
sbt:MyApi>

calling hello2 twice works, but is very slow

In my SJS version, it looks like the shutdown is what is slow. Both calls succeed immediately.

That was also my observation (on the JVM for me)
On Fri, Oct 14, 2022, 17:51 Arman Bilge @.***> wrote:

calling hello2 twice works, but is very slow

In my SJS version, it looks like the shutdown is what is slow. Both calls
succeed immediately.
—
Reply to this email directly, view it on GitHub
https://github.com/http4s/http4s/issues/4935#issuecomment-1279504538,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABH467FR6X2SZOF5H74Y4DTWDHIV3ANCNFSM47B6GXTQ
.
You are receiving this because you commented.Message ID:
@.***>

Ok, thank you everyone. I think this is a minimizer. IDK. It minimizes something, which seems bad 🙃
//> using scala "3.2.0"
//> using lib "org.http4s::http4s-ember-server::0.23.16"
//> using lib "org.http4s::http4s-ember-client::0.23.16"
//> using lib "org.http4s::http4s-blaze-server::0.23.12"
//> using lib "org.http4s::http4s-jdk-http-client::0.7.0"

import cats.effect.*
import org.http4s.*
import org.http4s.ember.server.*
import org.http4s.ember.client.*
import org.http4s.server.middleware.{Logger => ServerLogger}
import org.http4s.client.middleware.{Logger => ClientLogger}
import org.http4s.syntax.all.*
import org.http4s.blaze.server.*
import org.http4s.jdkhttpclient.*

object Bug extends IOApp.Simple:

  def routes = HttpRoutes.of[IO] { case _ =>
    IO(Response[IO]().withEntity("Hello"))
  }

  def app = ServerLogger(
    true,
    true,
    cats.arrow.FunctionK.id[IO],
    logAction = Some((x: String) => IO.println(s"server: $x"))
  )(routes.orNotFound)

  def emberServer =
    EmberServerBuilder.default[IO].withHttpApp(app).build

  def blazeServer =
    BlazeServerBuilder[IO].withHttpApp(app).resource

  def emberClient = EmberClientBuilder
    .default[IO]
    .build

  def jdkClient = JdkHttpClient.simple[IO]

  def client = emberClient
    .map(
      ClientLogger(
        true,
        true,
        logAction = Some((x: String) => IO.println(s"client: $x"))
      )(_)
    )

  def run =
    blazeServer.surround {
      client.use { client =>
        client
          .run(
            Request[IO](method = Method.POST, uri = uri"http://localhost:8080")
          )
          .use(_.as[String])
          .replicateA(2)
          .flatMap(IO.println)
      }
    }

tl;dr ember-client seems to have a problem sending chunked requests with empty bodies. This is what my reproducer above is doing, as is the smithy reproducer. I'm not sure if this is related to @channingwalton's problem?

Perhaps we need a request version of https://github.com/http4s/http4s/pull/6444. Which surprise, surprise was proposed there 😅

Hmmmm. So if I change the routes to
  def routes = HttpRoutes.of[IO] { case req =>
    req.as[Unit] *> IO(Response[IO]().withEntity("Hello"))
  }

then everything is hunky-dory again. This is forcing the server to drain the entity body.
So now my hypothesis is that there are 3ish issues here:

POSTing an empty Stream with ember-client gets sent with Transfer-Encoding: chunked. So the body consists of something like 0\r\n\r\n.
If the server is expecting an empty body, it does not drain the entity. This leaves the 0\r\n\r\n on the wire and mucks up the connection.
ember-client is unable to recover/retry this failure.

Assuming that my analysis above is correct:

This is fundamentally a server-side problem: even if the server is expecting an empty body, it must still drain the body bytes. So @daddykotex that suggests this is a bug in smithy4s after all 😛
When you don't control the server, a workaround is to explicitly set an empty entity body for requests e.g. .withEntityBody("") or .withEntityBody(ByteVector.empty) or something. Don't use Stream.empty.
ember-client can behave more "normally" by implementing something like https://github.com/http4s/http4s/pull/6444 to try and recognize empty streams and not use chunked transfer encoding for those. Also perhaps ember should be able to open a new connection and retry requests that encounter this error.

This is fundamentally a server-side problem: even if the server is expecting an empty body, it must still drain the body bytes. So @daddykotex that suggests this is a bug in smithy4s after all

Though we're certainly gonna amend our usage of http4s in smithy4s, doesn't this imply some slight design issue in http4s ? Wasn't there an idea floating at some point that HttpApp[IO] should actually de-alias to Kleisli[Resource[IO, ?], Request[IO], Response[IO]] or something ?

Yes, something like that is on the books for 1.0. See:

https://github.com/http4s/http4s/discussions/4846#discussioncomment-753677
https://github.com/http4s/http4s/pull/2663

I'm not sure if this is related to @channingwalton's problem?

I’m not sure either. In my case the client is making a request to a LoginRadius service, so I have no idea what that is. I’ll get the details of the request. It the response is expected to be Json.

By chance, I have stumbled across a fairly reliable test case for this but it still involves a third party service. But having something reliable might let me pinpoint what is going on.

@channingwalton feel free to try this snapshot with my fix from https://github.com/http4s/http4s/pull/6752.
https://s01.oss.sonatype.org/content/repositories/snapshots/org/http4s/http4s-ember-client_2.13/0.23.16-159-6821a48-SNAPSHOT/http4s-ember-client_2.13-0.23.16-159-6821a48-SNAPSHOT.pom

@channingwalton feel free to try this snapshot with my fix from #6752.
https://s01.oss.sonatype.org/content/repositories/snapshots/org/http4s/http4s-ember-client_2.13/0.23.16-159-6821a48-SNAPSHOT/http4s-ember-client_2.13-0.23.16-159-6821a48-SNAPSHOT.pom

Unfortunately, I am still seeing the issue but I am still not convinced that this is actually a genuine issue with the server. I am going to have to do more work on that.

I've spent a day trying to pin this down and got nowhere, except that reverting to blaze solves the issue.
I will try again tomorrow.

I've not managed to get much time on this. We've reverted to blaze for now and I'll keep trying ember releases.

This has occurred for us as well. I think it's the same as described here. In that case it's a problem with POST and PUT with empty bodies. Adding .withEntity(ByteVector.empty) as a workaround seems to work.

@flindskog thanks for reporting. You may also try this snapshot.
https://s01.oss.sonatype.org/content/repositories/snapshots/org/http4s/http4s-ember-client_2.13/0.23.16-159-6821a48-SNAPSHOT/http4s-ember-client_2.13-0.23.16-159-6821a48-SNAPSHOT.pom

@armanbilge Thank you!! That version works fine.

@flindskog thanks for reporting. You may also try this snapshot. https://s01.oss.sonatype.org/content/repositories/snapshots/org/http4s/http4s-ember-client_2.13/0.23.16-159-6821a48-SNAPSHOT/http4s-ember-client_2.13-0.23.16-159-6821a48-SNAPSHOT.pom

Hi, any idea when this is likely to be released?

Hi, any idea when this is likely to be released?

Yes, we will have a release early in the new year. It's been a minute since the last one 😅

Just started using the client with the OpenSearch on Amazon. Almost half of the requests (from thousands) ends up with this error. Using version 0.23.18

@michkhol thanks for reporting—can you provide any sort of reproducer? Or at least detailed logs of the full request/response? I'd really like to fix this one, but I need fresh examples :)

Well, re: OpenSearch, this is located in the internal corporate cloud, so no outside access. But it's a standard AWS service, anyone could use it with an account, I suppose. I don't know if it can be used for free though.
Re: detailed logs, how detailed should they be? This is the sending code: ```
req = POST(uri).withEntity(json.toString).withContentType(Content-Type(MediaType.application.json))
resp <- client.stream(req).evalMap { r => r.as[String].map(x => (r.status, x)) }.compile.toList.map(_.head)
and this is the error:

org.http4s.ember.client.internal.ClientHelpers$$anon$2: Remote Disconnect: Received zero bytes after sending request
at org.http4s.ember.client.internal.ClientHelpers$$anon$1.applyOrElse(ClientHelpers.scala:180)
at org.http4s.ember.client.internal.ClientHelpers$$anon$1.applyOrElse(ClientHelpers.scala:174)
at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:288)
at cats.ApplicativeError.recoverWith$$anonfun$1(ApplicativeError.scala:172)
at main$ @ gov.nih.nlm.ncbi.pmsindexer.PmdmSearchIndexer$.main(PmdmSearchIndexer.scala:65)
at timeout @ gov.nih.nlm.ncbi.pmdm.applog.AppLogger$.getRef(AppLogger.scala:90)
at timeoutTo @ org.http4s.ember.core.Util$.timeoutToMaybe(Util.scala:110)
at getAndSet @ org.typelevel.keypool.KeyPool$.destroy(KeyPool.scala:120)
at flatMap @ org.http4s.ember.client.internal.ClientHelpers$.writeRead$1$$anonfun$1(ClientHelpers.scala:167)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
Caused by: org.http4s.ember.core.EmberException$EmptyStream: Cannot Parse Empty Stream
at org.http4s.ember.core.EmberException$EmptyStream$.apply(EmberException.scala:46)
at org.http4s.ember.core.Parser$MessageP$.recurseFind$$anonfun$1$$anonfun$1(Parser.scala:55)
at main$ @ gov.nih.nlm.ncbi.pmsindexer.PmdmSearchIndexer$.main(PmdmSearchIndexer.scala:65)
at timeout @ gov.nih.nlm.ncbi.pmdm.applog.AppLogger$.getRef(AppLogger.scala:90)
at timeout @ gov.nih.nlm.ncbi.pmdm.applog.AppLogger$.getRef(AppLogger.scala:90)
at flatMap @ org.http4s.ember.core.Parser$MessageP$.recurseFind$$anonfun$1(Parser.scala:56)
at flatMap @ org.http4s.ember.core.Parser$MessageP$.recurseFind(Parser.scala:60)

Re: detailed logs, how detailed should they be?

It seems this is a problem with how ember is handling the data sent over the connection ("cannot parse empty stream"). So ideally, we need to see the entire response (i.e. the raw bytes) coming from the server, to understand why ember is having so much trouble with it.

Is that POST the only kind of request that is happening? Can you reproduce the error (at least sometimes) with only that single request, or do you need to make more than one?
The reason I ask is this:

Almost half of the requests (from thousands)

Since you say almost half, I wonder if the first request/response succeeds, but it corrupts the connection (e.g. leaving unprocessed bytes on the wire). So the, the second request fails with your error, and the connection is terminated. Then, the next request starts a brand new connection, and so on.

Yes, POST is the only one that's happening right now.

So ideally, we need to see the entire response
I think I can do that.

I wonder if the first request/response succeeds, but it corrupts the connection
It is certainly possible, and by looking at the exact timing it looks like that, every other request fails. But without seeing the actual connection ids I cannot be sure.

To make sure I switched to blaze and I had no issues, so it is definitely the ember client.
Re: entire response, I'm still new to http4s and I think I can get the body of the response as a byte array. But will it be "raw" enough? If you need the whole response (including headers, etc.) as a raw bytestream, I would need help to get that.

Thanks for checking! Yes, unfortunately we do need the whole response, prior to parsing.
OpenSearch is open source, so I still wonder if some kind of reproducer is feasible.

https://github.com/opensearch-project

If that is not something you are able to set up (thanks for all your help so far!), can you at least provide more details about which OpenSearch APIs you are calling?

It's  an indexing request, very simple, exactly from the docs for a single document: https://opensearch.org/docs/latest/opensearch/index-data/. It says PUT there, but seems like it accepts POST too. It will complain if the method is not right.

@michkhol great, thanks! We will try and reproduce it. Other details such as opensearch version and maybe how big your request bodies are would be helpful as well? In case that is relevant.

Adding a possibly important data point to the recent discussions.
I have been tracking this issue for a while and I have a somewhat convoluted way to reproduce this issue 100% of the time.
I discovered the issue while migrating one of library from Blaze to Ember as well.
Here is the migration PR https://github.com/agourlay/cornichon/pull/427
To reproduce the error:
1 - clone git@github.com:agourlay/cornichon.git
2 - move to branch http4s-ember-client
3 - within an SBT shell run testOnly *property*
You should witness 3 tests failing with Remote Disconnect: Received zero bytes after sending request.
There is a total of 4 tests spawning an HTTP server on the port 8080 one after the other while sharing a global HTTP client.
All tests pass successfully when run in isolation, but when run sequentially, only the first test is green while the three remaining ones are red.
It seems the globally shared client is left in a corrupted state after the first server is shutdown.
I apologize for the complicated reproducer, I am not super happy about asking people to dig in my code base but I thought it might help.

@agourlay thank you, will take a look!

@agourlay thanks, that was very helpful. I just wrote a summary of what is happening at https://github.com/agourlay/cornichon/pull/427#issuecomment-1435268368.
The tl;dr is that Ember caches connections and tries to re-use them. If the server has disappeared in the meantime, then upon sending a request it discovers that: Remote Disconnect: Received zero bytes after sending request. In the case of a GET request, it would retry in that situation, but a POST request is not idempotent, so it will not retry it since it may not be safe to do so. Hence, it escalates the error.
It's quite possible this is what's happening with OpenSearch as well.
So it's a good question: can Ember handle these situations more gracefully? What is e.g. Blaze doing?

Other details such as opensearch version and maybe how big your request bodies are would be helpful as well?
Here is the opensearch response

"version": {
        "number": "7.10.2",
        "build_type": "tar",
        "build_hash": "unknown",
        "build_date": "2022-11-15T05:29:22.155152Z",
        "build_snapshot": false,
        "lucene_version": "8.10.1",
        "minimum_wire_compatibility_version": "6.8.0",
        "minimum_index_compatibility_version": "6.0.0-beta1"
    },

The request bodies are about 4K each.

Thanks to everyone for the help! Here is a new, minimum reproducer. It kills and restarts the server between two POST requests, which is sufficient to trigger the problem.
//> using lib "org.http4s::http4s-ember-server::0.23.18"
//> using lib "org.http4s::http4s-ember-client::0.23.18"

import cats.effect._
import com.comcast.ip4s._
import org.http4s._
import org.http4s.syntax.all._
import org.http4s.ember.client.EmberClientBuilder
import org.http4s.ember.server.EmberServerBuilder
import scala.concurrent.duration._

object App extends IOApp.Simple {

  def run = {
    val req = Request[IO](Method.POST, uri = uri"http://localhost:8080")
    val server = EmberServerBuilder.default[IO].withShutdownTimeout(0.seconds).build
    EmberClientBuilder.default[IO].build.use { client =>
      server.surround(client.status(req)) *>
        server.surround(client.status(req)) *>
        IO.unit
    }
  }

}

org.http4s.ember.client.internal.ClientHelpers$$anon$2: Remote Disconnect: Received zero bytes after sending request
        at org.http4s.ember.client.internal.ClientHelpers$$anon$1.applyOrElse(ClientHelpers.scala:180)
        at org.http4s.ember.client.internal.ClientHelpers$$anon$1.applyOrElse(ClientHelpers.scala:174)
        at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:288)
        at cats.ApplicativeError.recoverWith$$anonfun$1(ApplicativeError.scala:172)
        at main$ @ App$.main(bug.scala:12)
        at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(Util.scala:104)
        at timeoutTo @ org.http4s.ember.core.Util$.timeoutToMaybe(Util.scala:110)
        at getAndSet @ org.typelevel.keypool.KeyPool$.destroy(KeyPool.scala:120)
        at flatMap @ org.http4s.ember.client.internal.ClientHelpers$.writeRead$1$$anonfun$1(ClientHelpers.scala:167)
        at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
Caused by: org.http4s.ember.core.EmberException$EmptyStream: Cannot Parse Empty Stream
        at org.http4s.ember.core.EmberException$EmptyStream$.apply(EmberException.scala:46)
        at org.http4s.ember.core.Parser$MessageP$.recurseFind$$anonfun$1$$anonfun$1(Parser.scala:55)
        at main$ @ App$.main(bug.scala:12)
        at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(Util.scala:104)
        at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(Util.scala:104)
        at flatMap @ org.http4s.ember.core.Parser$MessageP$.recurseFind$$anonfun$1(Parser.scala:56)
        at flatMap @ org.http4s.ember.core.Parser$MessageP$.recurseFind(Parser.scala:60)

Could be a possible solution to use cached connections only for guaranteed idempotency?

In general you want to cache connections as much as possible, because setting up new connections is expensive. Furthermore, ideally the server would also be preserving that connection, so everything should just work.
The problem is when the server drops that connection. I wonder if this is what OpenSearch is doing to you.
I'm looking at solutions for Ember Client, where it can try and find out if a connection has been dropped before beginning the request. In that case, it can start a new connection. The problem is after it has started the request, and then it discovers the connection has been dropped, then you run into idempotency issues.

Ok, I've now got a potential fix up in https://github.com/http4s/http4s/pull/6980 and I have published a snapshot of it. @channingwalton @etspaceman @michkhol @agourlay would you be able to give that a try and see if it fixes your issues? Thank you so much!
resolvers += "s01 snapshots" at "https://s01.oss.sonatype.org/content/repositories/snapshots/"
libraryDependencies += "org.http4s" %% "http4s-ember-client" % "0.23.18-98-66d0795-SNAPSHOT"

Good news, my tests are green using the provided snapshot artifacts :tada:
I am pretty excited to finally being able to complete that long standing Ember migration!
@armanbilge Thank you very much for the fantastic support on this issue :pray:

Thanks to you, your reproducer was a great lead! I am hoping we can finally close this one, let's see how the others fare :)

I will try it, 🤞🏼

I will try it over the weekend.

Awesome. I'll give this a try sometime today.

This version introduces a new non-deterministic error for me:
[error]     at org.http4s.ember.client.internal.ClientHelpers$.writeRead$1$$anonfun$1(ClientHelpers.scala:169)
	at clear @ org.http4s.client.middleware.Retry$.retryLoop$1(Retry.scala:90)
	at flatTap @ munit.CatsEffectFunFixtures$ResourceFixture$.apply$$anonfun$2$$anonfun$1(CatsEffectFunFixtures.scala:53)
	at clear @ org.http4s.client.middleware.Retry$.retryLoop$1(Retry.scala:90)
[error]     at defer @ org.http4s.ember.client.internal.ClientHelpers$.writeRead$1(ClientHelpers.scala:171)
[error]     at unsafeToFuture @ munit.CatsEffectFunFixtures$ResourceFixture$.apply$$anonfun$2$$anonfun$1(CatsEffectFunFixtures.scala:55)
[error]     at void @ org.typelevel.keypool.KeyPool$.reap(KeyPool.scala:187)
[error]     at unsafeToFuture @ munit.CatsEffectFunFixtures$ResourceFixture$.apply$$anonfun$2$$anonfun$1(CatsEffectFunFixtures.scala:55)
[error]     at *> @ org.http4s.client.middleware.Retry$.retryLoop$1(Retry.scala:133)
[error]     at timeout$extension @ org.http4s.ember.client.internal.ClientHelpers$.writeRead$1(ClientHelpers.scala:155)```

Example run:

https://github.com/etspaceman/kinesis4cats/actions/runs/4212916794/jobs/7312277567

@etspaceman thanks for trying that! Are you getting that error on tests that were previously failing due to this bug, or is this on tests that used to be passing? 😅

Tests that were previously failing due to this bug.

100K+ requests to OpenSearch have run without the error. So far so good.

Any idea when it's gonna be released? I could not find this pull request mentioned in any of the 0.23.19RC releases.

I'm aiming to get it released in 0.23.19 final, but not sure yet when exactly that will be. Hopefully in the next week or two.

Thanks a lot!

@michkhol this is now available in v0.23.19-RC3.


Fix & add tests

The current tests folder contains some early testing from library creation, nothing proper. Running tests currently fails with an import error, though the library works fine when used with a bundler.
Tests should be fixed so that they can run and some proper tests should be added to check functionality.

Done


Internal audio only

Can you make this app to capture the audio from the device like some screen recorders which can record only the device audio.

Can you make this app to capture the audio from the device like some screen recorders which can record only the device audio.

Do you mean like what VIRTUAL AUDIO CABLE do in Windows PC?
That would need me to write a new virtual sound driver, re-cooking the whole android firmware, then inject it back to phone. That is out of my capability, sorry 😁
https://groups.google.com/g/android-platform/c/A6omsAOnLLg
https://stackoverflow.com/questions/39746221/what-is-required-to-make-android-use-a-new-audio-hal


Fix various bugs

842e8b1 - roles/entitlement/tasks/inspect.yml: Get the state of the nodes: correctly call 'oc get nodes'

d807870 - roles/entitlement/tasks/test_wait.yml: stop waiting for Pod completion after 120s

828e095 - callback_plugins/human_log.py: correctly group 'task.get_path()' with its task logs

c1a6415 - roles/entitlement/tasks/test_wait.yml: retry '0' times when not waiting

f711e2f - roles/entitlement: retry 20 times when waiting

d62ed9c - toolbox/entitlement/test.sh: add '--no-inspect' flag

This flag allows muting the inspection tasks (eg, when we expect the
test to confirm that the entitlement is not deployed)

/test gpu-operator-e2e

I added a patch to increase the number of retries when waiting for the InstallPlan to appear, it failed in this test:
/opt/ci-artifacts/src/playbooks/roles/nv_gpu/tasks/install_nv.yml:49
Friday 05 March 2021  14:34:18 +0000 (0:00:00.772)       0:00:04.909 **********
TASK: nv_gpu : Find the GPU Operator OperatorHub InstallPlan
TASK: nv_gpu : Find the GPU Operator OperatorHub InstallPlan
changed: True

<command> oc get installplan -oname -loperators.coreos.com/gpu-operator-certified.openshift-operators -n openshift-operators

==> FAILED attempt #1/10

==> FAILED attempt #2/10

==> FAILED attempt #3/10

==> FAILED attempt #4/10

==> FAILED attempt #5/10

==> FAILED attempt #6/10

==> FAILED attempt #7/10

==> FAILED attempt #8/10

==> FAILED attempt #9/10

==> FAILED attempt #10/10

TASK: nv_gpu : Find the GPU Operator OperatorHub InstallPlan
changed: True

<command> oc get installplan -oname -loperators.coreos.com/gpu-operator-certified.openshift-operators -n openshift-operators

---


Adding custom validation directive removes all selected values

I added a custom validation directive to ui-select that validates all selected items to make sure each item has a valid email property.
But after validation returns falsy, all selected items gets lost:
http://plnkr.co/edit/gW52dvUjMx4ydHEaHa4a?p=preview

Add ng-model-options="{allowInvalid: true}" next to ng-model.
Reference: https://docs.angularjs.org/api/ng/directive/ngModelOptions

I'm also in need of a fix for this issue - and allowing invalid won't work in my situation. I'm researching the right way to handle this, if I find the answer I'll at minimum post here, or try to submit a pull request.

I've submitted a pull request for this issue: https://github.com/angular-ui/ui-select/pull/1359


use case: given points

I have this use case where I want to construct a polynomial where I have as input the secret and some additional predefined points. The idea is that each "application" (whatever that is) can present its own point(s), and that the secret remains the same, independent of the "application". Hence you can have a single AES encryption (key) shared by all "applications".

I can’t help you with that.

FYI... implemented this feature. Available at: https://github.com/e-Contract/shamir/tree/fcorneli

PR available at https://github.com/codahale/shamir/pull/14


WIP - fix #45 - add custom_file define

This is a WIP for the issue in #49 and how I am now deploying custom configs.
We could even hardcode the custom- in front.
Once the design is clear, I could add tests and fix documentation.

One think that could simplify all these individual file copies is we could just use a single file to copy the whole directory from preflight to live directory. Would remove a load of code.

One think that could simplify all these individual file copies is we could just use a single file to copy the whole directory from preflight to live directory. Would remove a load of code.

Oh that's actually an excellent idea and it would also purge things...

We have a successor #48


When moving folders in javascript -  imports messed up!

Hi,
I am desperate about this, how do people even work like this, with this happening.
The auto-refactor of imports worked for some time since you added this feature, but It started to make these problems some time ago and I see that it is not fixed in the releases so I create this issue.

Have not found similar issue

Insiders have the same issue

VS Code version: Code - Insiders 1.37.0-insider (fd5985489982fe14381dd56e6f0b29c53d5f991b, 2019-07-18T14:11:10.247Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item
Value

CPUs
AMD Ryzen 5 1600X Six-Core Processor            (12 x 3600)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwareoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: enabledskia_deferred_display_list: disabled_offskia_renderer: disabled_offsurface_synchronization: enabled_onvideo_decode: enabledviz_display_compositor: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.95GB (15.39GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions: none
Steps to Reproduce:

When I move any folder to anywhere, all absolute imports (even from node_modules) get added the path from the base till this folder. So I need to go and ctrl-z every file.
What actually should happen is that absolute imports should not change.
I have all my source in ./src.
And jsconfig.json is in ./

{
	"compilerOptions": {
		"baseUrl": "./src",
		"target": "ES6"
	},
	"exclude": [
		"**/node_modules/**",
		"**/dist/**",
		"**/db/**",
		"**/.history/**",
		"**/.git/**",
		"**/not_used/**",
	]
}

Beneath is the folder root
Here are screenshots before and after

Does this issue occur when all extensions are disabled?: Yes/No
Yes, all extensions are disabled

Can you please share a small example project that demonstrates this issue

Thanks for taking care!
Demo repo for this issue:
https://github.com/lishine/demonstrate-vscode-issue-77635
For example, try to move  folder 'common' into folder '1'.
The problem is still reproduced in Insiders without extensions.

Same problem when renaming folder

@mjbvz
I provided the info.
Please do something.
The auto refactor of imports is still non functional...
Is there anybody there using javascript :) ? Please support it...

It looks like the example project is not complete so it is difficult to understand what the expected behavior is supposed to be. For example 'common/utils/utils' does not exist.
Try adding "checkJs": true to your jsconfig to make sure the imports are being resolved properly in the first place

Adding this to jsconfig.json solves the problem.
I appreciate your help! :)
"moduleResolution": "node",

BTW, checkJs helped to see the imports resolve problem, but otherwise it shows many typescript issues, so it cannot be enabled all time for js dev, pity.


NEL benchmark: update spaCy version

Description

Recent commits show CI failures caused by wikid importing InMemoryLookupKB.  This is unexpected, as this error has been accounted for in wikid already.
The NEL benchmark itself hasn't been adjusted accordingly yet anyway, so this PR aims to support spacy>=3.0.0,<3.6.0 by relaxing the requirement in the project file and importing the correct KB class contingent on the spaCy version.
Types of change

Fix.
Checklist

[x] I confirm that I have the right to submit this contribution under the project's MIT license.
[x] I ran the tests, and all new and existing tests passed.
[x] I ran the update scripts in the .github folder, and all the configs and docs are up-to-date.

I think you missed requirements.txt, see: https://dev.azure.com/explosion-ai/Public/_build/results?buildId=24882&view=logs&j=f1d52934-b974-5ece-337e-e658a4411382&t=25745add-d8f1-5bdf-c7de-2e51e93603d8&l=20

I think you missed requirements.txt, see: https://dev.azure.com/explosion-ai/Public/_build/results?buildId=24882&view=logs&j=f1d52934-b974-5ece-337e-e658a4411382&t=25745add-d8f1-5bdf-c7de-2e51e93603d8&l=20

Thanks for the heads up! PR.


Merge xxx-browserify or not?

Should the xxx-browserify modules and authors merged into this package?
Pro:

less fragmentation
more maintainers for all node buildin libs
easier sharing of small common fragments, i.e. shims
easier to maintain

Contra:

less modular
"holy fuck. why are there 20 files and 1000s of lines of code in one repo"

Problems merging:

xxx-browserify modules could not longer be used alone
they can transform into wrappers for this package

Problems splitting:

duplicated effort to have each module tested in testling
no aggregated test results
repo for aggregating results
badge wouldn't update

Can you pipe webpack to testling instead of browserify?


Infinite Queries: wrong auto refetching with manual pageParams

Describe the bug
Infinite Queries can be used in two ways:

they implement getNextPageParam / getPreviousPageParam (automatic)
they pass pageParam directly to fetchNextPage / fetchPreviousPage (manual)

Mixing those two leads to a weird behaviour.
Your minimal, reproducible example
https://codesandbox.io/s/dazzling-lehmann-zibq9m?file=/pages/index.js
Steps to reproduce

scroll down, click on load newer
now, click the skip button
you now have projects 0-9 and 50-54 on screen. The stored pageParams are:
[0, 5, 50]
trigger an automatic refetch by re-focussing the window

--> you now have projects 0-14 on screen
pageParmas are:
[0, 5, 10]
Expected behavior
data on screen should not change / stay as it was initially fetched, and pageParams shouldn't change.
How often does this bug happen?
Every time
Screenshots or Videos
No response
Platform
all
react-query version
4
TypeScript version
No response
Additional context
The problem seems to be that during an automatic refetch, getNextParam is always invoked when it exists. We have no knowledge of the fact that the 3rd page was fetched with the "manual" approach.
It seems that mixing both approaches is not supported at all, even though there is an extra section in the docs that does exactly that:
https://tanstack.com/query/v4/docs/guides/infinite-queries#what-if-i-need-to-pass-custom-information-to-my-query-function
when we refetch pages, we determine if we are in manual mode or not by the presence of the getNextPageParam function:
https://github.com/TanStack/query/blob/d5c4330810c26fde7ed7219940216be97ef3867a/packages/query-core/src/infiniteQueryBehavior.ts#L122
But that function will always be present, even when a single page fetch was "overwritten" with a manual fetch.

One way to fix this would be to extend pageParams (or the whole infinite query data structure) by the information if the fetch happened initially as manual or not.
Actually, we only store pageParams for refetches when everything is using the manual approach. Put another way: We never access pageParams if a getNextPageParam function is present.

One way to fix this would be to extend pageParams (or the whole infinite query data structure) by the information if the fetch happened initially as manual or not.

@TkDodo Wouldn't that be a breaking change because pageParams is stored as an array right now

@TkDodo Wouldn't that be a breaking change because pageParams is stored as an array right now

yes, could be something for v5. Or, we extend the structure to be like:
pages: [{..}, {..}, {..}],
pageParams: [undefined, 5, 50],
manual: [false, false, true]

but this seems overly verbose and unnecessary.
One question is: Why even store pageParams if we didn't fetch manually? I think we could also just store pageParams IF we fetched manually and use undefined for other cases, so in the above example:
pages: [{..}, {..}, {..}],
pageParams: [undefined, undefined, 50],

if we encounter undefined, we would just invoke getNextPageParam (except for the first page, which always needs undefined passed), and if we have a stored value, we would use that.

Just tried this approach out and it doesn't work if you use bi-directional queries. Like, fetch first page, then fetchNext, then fetchPrevious. You would have 3 pages and all of them would have pageParam: undefined. The refetch then starts at the first entry in the array, which is the "previous page", so we'd need that value's pageParam from the fetch as well.
So my statement that we "only use the stored pageParams when we did manual fetching" was wrong ...
This seems to be pretty hard to fix conceptually 😅

@TkDodo  Okay then how about we go with your initial idea of using an array of objects, with the manual key and keep the pageParams as well for backwards compatibility. Something like this:
pages: [{..}, {..}, {..}],
pageParams: [undefined, 5, 50],
pageParamsWithManual: [{ page: undefined, manual: false }, { page: 5, manual: false }, { page: 50, manual: true }]

Internally we use the pageParamsWithManual (can be named something better) and rely on the manual key when refetching. With v5 deprecate the array format and use the new format for pageParams. This should do it right?

I think even extending the object with another entry could break something, for example, if you implemented optimistic updates without spreading:
queryClient.setQueryData(
  key,
  (prev) => prev ? { pages: update(prev.pages), pageParams: prev.pageParams } : prev
)

this is legit code in v4, as it returns an object with pages and pageParams, but it would take away pageParamsWithManual.
I think we should wait with this change for v4 and do your proposed change there.

Hey @TkDodo Since we have started working on v5 can I take up this issue and fix it for v5?

I'll write up a dedicated RFC for infinite queries in v5 soon. It will impact this issue so let's please hold off for now

PR:

#5004


link to openshift docs

On Fri, Dec 18, 2015 at 12:35:35PM -0800, Ben Parees wrote:

https://github.com/openshift/sti-perl/pull/65
-- Commit Summary --

link to openshift docs

-- File Changes --
M 5.16/README.md (2)

What about 5.20/README.md? Is the text applicable to 5.16 only?
-- Petr

@ppisar actually i updated the wrong readme.  i've been trying to just update the top-level readme in all the repos but i touched the wrong one in this case.  Thanks for catching it.
fixing here:
https://github.com/openshift/sti-perl/pull/66


SOLVCON build failed with the latest pybind11 (1.9.dev0)

Steps to Reproduce

Fetch the latest SOLVCON source
Make sure your build environment use the latest pip source. You may issue this command to install the latest pybind11

pip install -U https://github.com/pybind/pybind11/archive/master.zip

Build the code by

python setup.py build_ext --inplace

Expected Result
Build completed
Actuall Result
Error message
solvcon/march.cpp:128:20: note: in definition of macro âDECL_MARCH_PYBIND_CLASS_METHODâ
     wrapper_type & METHOD(Args&&... args) { \
                    ^
solvcon/march.cpp:133:36: error: âWrapBase<Wrapper, Wrapped, Holder>::wrapper_type& WrapBase<Wrapper, Wrapped, Holder>::def(Args&& ...) [with Args = {const char (&)[13], WrapUnstructuredBlock<NDIM>::pickle() [with long unsigned int NDIM = 3ul; WrapUnstructuredBlock<NDIM>::wrapper_type = WrapUnstructuredBlock<3ul>]::__lambda187}; Wrapper = WrapUnstructuredBlock<3ul>; Wrapped = march::UnstructuredBlock<3ul>; Holder = std::shared_ptr<march::UnstructuredBlock<3ul> >; WrapBase<Wrapper, Wrapped, Holder>::wrapper_type = WrapUnstructuredBlock<3ul>]â, declared using local type âWrapUnstructuredBlock<NDIM>::pickle() [with long unsigned int NDIM = 3ul; WrapUnstructuredBlock<NDIM>::wrapper_type = WrapUnstructuredBlock<3ul>]::__lambda187â, is used but never defined [-fpermissive]
     DECL_MARCH_PYBIND_CLASS_METHOD(def)
                                    ^
solvcon/march.cpp:128:20: note: in definition of macro âDECL_MARCH_PYBIND_CLASS_METHODâ
     wrapper_type & METHOD(Args&&... args) { \
                    ^
cc1plus: all warnings being treated as errors
error: Command "g++ -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilibmarch/include -I/home/tai271828/my-local-installation/miniconda3/solvcon-dev/include/python3.5m -I/home/tai271828/my-local-installation/miniconda3/solvcon-dev/include -I/home/tai271828/my-local-installation/miniconda3/solvcon-dev/lib/python3.5/site-packages/numpy/core/include -I/home/tai271828/my-local-installation/miniconda3/solvcon-dev/include/python3.5m -c solvcon/march.cpp -o build/temp.linux-x86_64-3.5/solvcon/march.o -Wall -Wextra -Werror -std=c++11 -Wno-unused-function -Wno-unreachable-code -Wno-sign-compare" failed with exit status 1

You may refer to this travis build for comprehensive build log https://travis-ci.org/tai271828/solvcon/jobs/185798179
Additional Info

SOLVCON source (commit 1063faf1943d1fc299c4b07bd8f56f895e6fca39)

pybind11 info

$ pip show pybind11
---
Metadata-Version: 1.1
Name: pybind11
Version: 1.9.dev0
Summary: Seamless operability between C++11 and Python
Home-page: https://github.com/wjakob/pybind11
Author: Wenzel Jakob
Author-email<EMAIL_ADDRESS>License: BSD
Location: /home/tai271828/my-local-installation/miniconda3/solvcon-dev/lib/python3.5/site-packages
Requires:
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: Utilities
  Programming Language :: C++
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.2
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  License :: OSI Approved :: BSD License

Thanks for the report.  I've been too long before I update my local pybind11.  Working on it.

Something seems wrong with the latest (2.0rc1) pybind11 pickling.  Still investigating.

A resolution is under review.


Error: 'java.security.PublicKey java.security.KeyPair.getPublic()' on a null object reference

I am getting this exception while trying to create a session between 2 users

Hi @kaveeshkanwal, could you please post a stack trace?

Hi @gpolitis ,
Thanks for the response.
I am using this library in an android chatting project.
I am getting the above mentioned exception in AuthContextImpl.java line number 106
SignatureM m = new SignatureM((DHPublicKey) getLocalDHKeyPair()
.getPublic(), getRemoteDHPublicKey(),
getLocalLongTermKeyPair().getPublic(), //line 106
getLocalDHKeyPairID());
public KeyPair getLocalLongTermKeyPair() throws OtrException {
if (localLongTermKeyPair == null) {
localLongTermKeyPair = getSession().getLocalKeyPair(); // THIS RETURNS NULL
}
return localLongTermKeyPair;
}
While debugging i have seen that both private and public keys get generated, but the above code is giving me a NPE.
I am still trying and debugging the code, but it would be great if you could help me regarding this issue.

Hi @kaveeshkanwal, this exception happens because you do not provide your own implementation of EngineHost.
To clarify, there are multiple types of keys. otr4j manages the session keys that are used while you're setting up an OTR session. However, there is also a long-term keypair that the chat client uses to authenticate, i.e. show that it is the same person as all the other times you chatted. For this, a long-term keypair is used. otr4j does not save/generate this keypair itself, but instead asks for it at the EngineHost implementation. (This EngineHost represents the chat software in which otr4j is embedded.)
In order to fix your problem, you would need to:

Implement your own EngineHost according to the interface definition.
Upon request provide (either restore from file or generate on the fly) your long-term keypair. (Method call is 'getLocalLongTermKeyPair()'.)

You generate this keypair only once on a single computer/user account, then store if for all future uses. otr4j relies on this long-term keypair to remain the same. That's how your chat partner is able to identify that you are the same person he chatted with last time.