Unable to set Nest thermostat temperature

Home Assistant release (hass --version):
0.28.1
Python release (python3 --version):
3.4.2
Component/platform:
climate / Nest
Description of problem:
Not able to set the temperature via automation rule.
Expected:
Temperature set to the data defined by "temperature".
Problem-relevant configuration.yaml entries and steps to reproduce:
automation.yaml
- alias: "TV Off - Change Temperature"
  trigger:
    platform: state
    entity_id: media_player.family_room_receiver
    to: 'off'
    from: 'on'
  condition:
    condition: state
    entity_id: input_boolean.guest_mode
    state: 'off'
  action:
    service: climate.set_temperature
    entity_id: climate.basement
    data:
      temperature: '78'

Traceback (if applicable):
Sep 12 21:56:10 automation hass[27493]: INFO:homeassistant.components.automation:Executing TV On - Lights
Sep 12 21:56:32 automation hass[27493]: INFO:homeassistant.components.automation:Executing TV Off - Lights
Sep 12 21:56:32 automation hass[27493]: INFO:homeassistant.components.automation:Executing TV Off - Change Temperature
Sep 12 21:56:33 automation hass[27493]: ERROR:homeassistant.core:BusHandler:Exception doing job
Sep 12 21:56:33 automation hass[27493]: Traceback (most recent call last):
Sep 12 21:56:33 automation hass[27493]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 865, in job_handler
Sep 12 21:56:33 automation hass[27493]: func(*args)
Sep 12 21:56:33 automation hass[27493]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 730, in _execute_service
Sep 12 21:56:33 automation hass[27493]: service(call)
Sep 12 21:56:33 automation hass[27493]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 586, in __call__
Sep 12 21:56:33 automation hass[27493]: self.func(call)
Sep 12 21:56:33 automation hass[27493]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/climate/__init__.py", line 240, in temperature_set_service
Sep 12 21:56:33 automation hass[27493]: climate.set_temperature(**kwargs)
Sep 12 21:56:33 automation hass[27493]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/climate/nest.py", line 145, in set_temperature
Sep 12 21:56:33 automation hass[27493]: self.device.target = temperature
Sep 12 21:56:33 automation hass[27493]: File "/home/hass/.homeassistant/deps/nest/nest.py", line 499, in target
Sep 12 21:56:33 automation hass[27493]: self._set('shared', data)
Sep 12 21:56:33 automation hass[27493]: File "/home/hass/.homeassistant/deps/nest/nest.py", line 278, in _set
Sep 12 21:56:33 automation hass[27493]: response.raise_for_status()
Sep 12 21:56:33 automation hass[27493]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/requests/models.py", line 862, in raise_for_status
Sep 12 21:56:33 automation hass[27493]: raise HTTPError(http_error_msg, response=self)
Sep 12 21:56:33 automation hass[27493]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://czfe15-front01-iad01.transport.home.nest.com/v2/put/shared.09AA01AC05160JEA

Additional info:

I believe you have to set target_temp_high in cool mode instead of temperature. this was updated in .28

I encountered the same problem even when I tried to set the temperature using webpage.
I played with the config and python-nest a bit and found out hass doesn't convert the temperature to Celsius before calling python-nest. The type conversion was removed in 0.28.
The problem only happens when you use Fahrenheit in hass config.

@justweb1 I tried using "target_temp_high" in my automation.yaml:
- alias: "TV Off - Change Temperature"
  trigger:
    platform: state
    entity_id: media_player.family_room_receiver
    to: 'off'
    from: 'on'
  condition:
    condition: state
    entity_id: input_boolean.guest_mode
    state: 'off'
  action:
    service: climate.set_temperature
    entity_id: climate.basement
    data:
      target_temp_high: '78'

And I got this error at reload of automation.yaml:
16-09-13 06:50:33 homeassistant.core: Invalid service data for climate.set_temperature: some but not all values in the same group of inclusion 'temperature' @ data[<temperature>]. Got None

I get the same error when I try to run from the developer tools:
Sep 13 06:57:52 automation hass[27493]: ERROR:homeassistant.core:Invalid service data for climate.set_temperature: some but not all values in the same group of inclusion 'temperature' @ data[<temperature>]. Got None

@gegao Now that you mention Celsius, I also notice that in the image that I posted in my first post shows the unit of measurement as Celsius.  My Nest app and Nest is using Fahrenheit and showing Fahrenheit everywhere.  Not sure why Home Assistant believes I'm using Celsius...

Celsius is the default and it converts to Fahrenheit. It should take this from you config and set properly. @turbokongen can you take a look at this?

This is a issue with the python-nest library only accepting celsius values. We are passing in fahrenheit.

Could any of you please test bugfix?

@turbokongen is the bugfix in the dev branch?  I'll test just want to make sure first!

https://github.com/home-assistant/home-assistant/issues/3373
You need to apply the PR to your code.

Not sure if I did it the correct way or not, but I simply copy/pasted the new code to mine and restarted.
New error at line 149 now:
Sep 13 10:32:44 automation hass[24740]: ERROR:homeassistant.core:BusHandler:Exception doing job
Sep 13 10:32:44 automation hass[24740]: Traceback (most recent call last):
Sep 13 10:32:44 automation hass[24740]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 865, in job_handler
Sep 13 10:32:44 automation hass[24740]: func(*args)
Sep 13 10:32:44 automation hass[24740]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 730, in _execute_service
Sep 13 10:32:44 automation hass[24740]: service(call)
Sep 13 10:32:44 automation hass[24740]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 586, in __call__
Sep 13 10:32:44 automation hass[24740]: self.func(call)
Sep 13 10:32:44 automation hass[24740]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/climate/__init__.py", line 240, in temperature_set_service
Sep 13 10:32:44 automation hass[24740]: climate.set_temperature(**kwargs)
Sep 13 10:32:44 automation hass[24740]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/climate/nest.py", line 149, in set_temperature
Sep 13 10:32:44 automation hass[24740]: self.device.target = temperature
Sep 13 10:32:44 automation hass[24740]: File "/home/hass/.homeassistant/deps/nest/nest.py", line 499, in target
Sep 13 10:32:44 automation hass[24740]: self._set('shared', data)
Sep 13 10:32:44 automation hass[24740]: File "/home/hass/.homeassistant/deps/nest/nest.py", line 278, in _set
Sep 13 10:32:44 automation hass[24740]: response.raise_for_status()
Sep 13 10:32:44 automation hass[24740]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/requests/models.py", line 862, in raise_for_status
Sep 13 10:32:44 automation hass[24740]: raise HTTPError(http_error_msg, response=self)
Sep 13 10:32:44 automation hass[24740]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://czfe14-front01-iad01.transport.home.nest.com/v2/put/shared.09AA01AC05160JEA

Ok. I updated the PR, I think i mixed up from and to type of the units also.
Enable logger in your install, and watch the logs for entries when setting temp.

Looks like you fixed it!  Thanks and I look forward to seeing this in a future bugfix.
I did put it in debug and if you're interested here is related output:
Sep 13 13:03:06 automation hass[25692]: INFO:homeassistant.components.http:Serving /api/services/climate/set_temperature to <IP_ADDRESS> (auth: True)
Sep 13 13:03:06 automation hass[25692]: INFO:homeassistant.core:Bus:Handling <Event call_service[L]: service_data=temperature=74, entity_id=climate.basement, service=set_temperature, service_call_id=1978722032-10, domain=climate>
Sep 13 13:03:06 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> FORWARDING <Event call_service[L]: service_data=temperature=74, entity_id=climate.basement, service=set_temperature, service_call_id=1978722032-10, domain=climate>
Sep 13 13:03:06 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> WRITING data: {"data": {"service_data": {"temperature": "74", "entity_id": "climate.basement"}, "service": "set_temperature", "service_call_id": "1978722032-10", "domain": "climate"}, "event_type": "call_service", "origin": "LOCAL", "time_fired": "2016-09-13T17:03:06.358015+00:00"}
Sep 13 13:03:06 automation hass[25692]: DEBUG:homeassistant.components.climate.nest:Nest set_temperature-input-value=23.3
Sep 13 13:03:06 automation hass[25692]: DEBUG:homeassistant.components.climate.nest:Nest set_temperature-output-value=23.3
Sep 13 13:03:06 automation hass[25692]: DEBUG:requests.packages.urllib3.connectionpool:"POST /v2/put/shared.09AA01AC05160JEA HTTP/1.1" 200 0

Sep 13 13:03:11 automation hass[25692]: INFO:homeassistant.core:Bus:Handling <Event state_changed[L]: old_state=<state climate.basement=unknown; unit_of_measurement=°C, friendly_name=Thermostat Control, humidity=48, current_temperature=78.0, target_humidity=35.0, target_temp_low=78.0, target_temp_high=78.0, max_temp=90.0, entity_picture=https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png, min_temp=50.0, temperature=78.0, mode=cool, away_mode=off @ 2016-09-13T12:52:27.526248-04:00>, new_state=<state climate.basement=unknown; unit_of_measurement=°C, friendly_name=Thermostat Control, humidity=48, current_temperature=78.0, target_humidity=35.0, target_temp_low=74.0, target_temp_high=74.0, max_temp=90.0, entity_picture=https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png, min_temp=50.0, temperature=74.0, mode=cool, away_mode=off @ 2016-09-13T12:52:27.526248-04:00>, entity_id=climate.basement>
Sep 13 13:03:11 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> FORWARDING <Event state_changed[L]: old_state=<state climate.basement=unknown; unit_of_measurement=°C, friendly_name=Thermostat Control, humidity=48, current_temperature=78.0, target_humidity=35.0, target_temp_low=78.0, target_temp_high=78.0, max_temp=90.0, entity_picture=https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png, min_temp=50.0, temperature=78.0, mode=cool, away_mode=off @ 2016-09-13T12:52:27.526248-04:00>, new_state=<state climate.basement=unknown; unit_of_measurement=°C, friendly_name=Thermostat Control, humidity=48, current_temperature=78.0, target_humidity=35.0, target_temp_low=74.0, target_temp_high=74.0, max_temp=90.0, entity_picture=https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png, min_temp=50.0, temperature=74.0, mode=cool, away_mode=off @ 2016-09-13T12:52:27.526248-04:00>, entity_id=climate.basement>
Sep 13 13:03:11 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> WRITING data: {"data": {"old_state": {"state": "unknown", "last_updated": "2016-09-13T16:52:27.526248+00:00", "attributes": {"max_temp": 90.0, "humidity": 48, "current_temperature": 78.0, "temperature": 78.0, "friendly_name": "Thermostat Control", "target_temp_low": 78.0, "target_temp_high": 78.0, "unit_of_measurement": "\u00b0C", "entity_picture": "https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png", "min_temp": 50.0, "target_humidity": 35.0, "mode": "cool", "away_mode": "off"}, "last_changed": "2016-09-13T16:52:27.526248+00:00", "entity_id": "climate.basement"}, "new_state": {"state": "unknown", "last_updated": "2016-09-13T17:03:11.153119+00:00", "attributes": {"max_temp": 90.0, "humidity": 48, "current_temperature": 78.0, "temperature": 74.0, "friendly_name": "Thermostat Control", "target_temp_low": 74.0, "target_temp_high": 74.0, "unit_of_measurement": "\u00b0C", "entity_picture": "https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png", "min_temp": 50.0, "target_humidity": 35.0, "mode": "cool", "away_mode": "off"}, "last_changed": "2016-09-13T16:52:27.526248+00:00", "entity_id": "climate.basement"}, "entity_id": "climate.basement"}, "event_type": "state_changed", "origin": "LOCAL", "time_fired": "2016-09-13T17:03:11.153292+00:00"}
Sep 13 13:03:11 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> FORWARDING <Event state_changed[L]: old_state=<state climate.basement=unknown; unit_of_measurement=°C, friendly_name=Thermostat Control, humidity=48, current_temperature=78.0, target_humidity=35.0, target_temp_low=78.0, target_temp_high=78.0, max_temp=90.0, entity_picture=https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png, min_temp=50.0, temperature=78.0, mode=cool, away_mode=off @ 2016-09-13T12:52:27.526248-04:00>, new_state=<state climate.basement=unknown; unit_of_measurement=°C, friendly_name=Thermostat Control, humidity=48, current_temperature=78.0, target_humidity=35.0, target_temp_low=74.0, target_temp_high=74.0, max_temp=90.0, entity_picture=https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png, min_temp=50.0, temperature=74.0, mode=cool, away_mode=off @ 2016-09-13T12:52:27.526248-04:00>, entity_id=climate.basement>
Sep 13 13:03:11 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> WRITING data: {"data": {"old_state": {"state": "unknown", "last_updated": "2016-09-13T16:52:27.526248+00:00", "attributes": {"max_temp": 90.0, "humidity": 48, "current_temperature": 78.0, "temperature": 78.0, "friendly_name": "Thermostat Control", "target_temp_low": 78.0, "target_temp_high": 78.0, "unit_of_measurement": "\u00b0C", "entity_picture": "https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png", "min_temp": 50.0, "target_humidity": 35.0, "mode": "cool", "away_mode": "off"}, "last_changed": "2016-09-13T16:52:27.526248+00:00", "entity_id": "climate.basement"}, "new_state": {"state": "unknown", "last_updated": "2016-09-13T17:03:11.153119+00:00", "attributes": {"max_temp": 90.0, "humidity": 48, "current_temperature": 78.0, "temperature": 74.0, "friendly_name": "Thermostat Control", "target_temp_low": 74.0, "target_temp_high": 74.0, "unit_of_measurement": "\u00b0C", "entity_picture": "https://d3rnbxvnd0hlox.cloudfront.net/images/channels/811416297/icons/large.png", "min_temp": 50.0, "target_humidity": 35.0, "mode": "cool", "away_mode": "off"}, "last_changed": "2016-09-13T16:52:27.526248+00:00", "entity_id": "climate.basement"}, "entity_id": "climate.basement"}, "event_type": "state_changed", "origin": "LOCAL", "time_fired": "2016-09-13T17:03:11.153292+00:00"}
Sep 13 13:03:11 automation hass[25692]: INFO:homeassistant.core:Bus:Handling <Event service_executed[L]: service_call_id=1978722032-10>
Sep 13 13:03:11 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> FORWARDING <Event service_executed[L]: service_call_id=1978722032-10>
Sep 13 13:03:11 automation hass[25692]: DEBUG:homeassistant.components.api:STREAM<PHONE_NUMBER> WRITING data: {"data": {"service_call_id": "1978722032-10"}, "event_type": "service_executed", "origin": "LOCAL", "time_fired": "2016-09-13T17:03:11.411295+00:00"}

Nice, so you have Fahrenheit as unit in HomeAssistant, right?
Can anyone with Celsius as unit test, or maybe you can just switch unit and test? @jwl17330536

Yes, I had Fahrenheit as my unit of measure.  I restarted with metric and it worked with that too.

Perfect! Thank you for testing! :bow:
Closed by #3373


Canvas Added to Now Playing View

Canvas is now playable on the new "Now Playing" view. The changes made to the code base are minimal.
The player is cleared and created again when switching between full screen and windowed mode.
Small quality change to the code by replacing querySelectorAll with querySelector, as only the first element was being used.

If you could update the README accordingly as well, it does mention the fullscreen view...

Maybe a switch (boolean flag) at the top to switch between the two modes would be ideal

Oh wait I see, both are supported. A flag to disable either, then

@itsmeow I added the config object where you can decide which view you want, both are enabled by default. I also updated the readme file with the configuration topic.


Silent flag for when nothing is done

Example currently, when there are no changes to be made:
❯ yarn -s pretty-quick
🔍  Finding changed files since git revision 2b1de96.
🎯  Found 4 changed files.
✅  Everything is awesome!

I propose there be a cli flag --silent-noop that, when nothing is done, nothing is printed.
Would you accept a PR for this?

Might just require a bigger change sounds like

I don't really want to have to buffer up the logs

Ok, np, I can live with current behaviour.


Eureka install + bookinfo files

What this PR does / why we need it:
Adds Eureka install and bookinfo example files for Docker environment. Uses a fork of registrator to mirror the Docker registry to Eureka: https://github.com/ijsnellf/registrator
Which issue this PR fixes: contributes to https://github.com/istio/istio.github.io/issues/588
Special notes for your reviewer: corresponding PR to the docs forthcoming.
Release note:

Adds Eureka adapter example for Docker environment.

/lgtm

/lgtm

/lgtm

/lgtm

/lgtm

/lgtm on others behalf

/lgtm


Cups

1⃣ Is this something that is wanted/needed? Did you create an issue / discussion about it first?
Should solve #74, #81
2⃣ Does it contain multiple, unrelated changes? Please separate the PRs out.
Nope.
3⃣ Does it include tests, if possible? (Not a deal-breaker, just a nice-to-have)
Currently I modified the existing tests, will add more if needed.
4⃣ Please include a thorough description of the improvement and reasons why it's useful.
Removes the smalot/cups-ipp dependency.
5⃣ Thanks for contributing! 🙌
Some side notes.

Currently there is no attribute filtering when querying for printers.
Currently you can only send 1 file/text per job. IPP actually allows multiple prints per job.
I couldn't test against a password protected cups server.

Thank you @vatsake, it fixes #74 for me!
@rawilk, will you consider approving this PR?
I'm willing to help to do some commits rebasing/polishing if needed.


Text color doesn't sync with theme

Quite simple. Added white fill in the set rule, that's how it looks

fck my bad. Won't sync, but I put the fill in the wrong line lul


Problems with Gromacs parser from umbrella sampling simulation

A user posted some issues with the Gromacs parser for a pair of umbrella sampling simulations. It is unclear what the issue is upon first glance (probably there are several issues). Will debug here...

First problem: str_to_energies function is not properly parsing the log file. str_to_energies function has this "P" in the regex matcher, what is this for? What happens is that in the example data #2 it is leaving out "P" whenever it begins a key word (possibly only follow by a period, I am not sure). If I remove the P and just require 2 spaces after the end of the string, I get the expected behavior.
@ladinesa  Is this correct? I don't know what the original intention was here. (You might want to look at the rest of the str_to_energies function
def str_to_energies(val_in):
  energy_keys_re = re.compile(r'(.+?)(?:  |\Z| P)')  # CURRENT
  energy_keys_re = re.compile(r'(.+?)(?:  |\Z|  )')  # SUGGESTED

 Example data #1:

    Energies (kJ/mol)
        G96Bond       G96Angle  Improper Dih.        LJ (SR)        LJ (LR)
    1.34229e+01    1.35642e+01    5.09136e+00    4.40070e+03   -7.67386e+01
  Disper. corr.   Coulomb (SR)   Coul. recip.      Potential    Kinetic En.
   -5.50745e+01   -2.50242e+04   -2.67009e+03   -2.33933e+04    3.69409e+03
   Total Energy    Temperature Pres. DC (bar) Pressure (bar)    dVremain/dl
   -1.96992e+04    2.91821e+02   -1.21477e+02   -2.75978e+02   -9.93820e+01

 Example data #2:

    Energies (kJ/mol)
           Bond            U-B    Proper Dih.  Improper Dih.      CMAP Dih.
    3.95020e+03    1.15459e+04    1.35009e+04    6.79906e+02   -3.09901e+02
          LJ-14     Coulomb-14        LJ (SR)  Disper. corr.   Coulomb (SR)
    3.17579e+03    7.45808e+04    2.23587e+05   -1.21064e+04   -2.46824e+06
   Coul. recip. Position Rest.   COM Pull En.      Potential    Kinetic En.
    6.83743e+03    6.02819e-04    3.23664e-12   -2.14279e+06    4.13386e+05
   Total Energy  Conserved En.    Temperature Pres. DC (bar) Pressure (bar)
   -1.72941e+06   -1.72950e+06    3.14117e+02   -1.31770e+02    2.11987e+02
   Constr. rmsd
    3.63345e-06

Above suggestion was not sufficient, fix by Alvin:
energy_keys_re = re.compile(r'(.+?)(?:  |\Z| (?=P))')

Ok, this was actually more complicated then we initially thought. The difficulty is dealing with single spaces between keys when the following key reaches the maximum character limit (14). For the record, here is the new code which addresses the issue:
        def str_to_energies(energies_block):

            max_char_name = 14
            rows = energies_block.strip().split("\n")
            names = [rows[i_row] for i_row in range(len(rows)) if i_row % 2 == 0]
            values = [rows[i_row] for i_row in range(len(rows)) if i_row % 2 != 0]

            names_re = r"(?<!\S)[A-Za-z0-9.()\/-]{1,}(?: [\S][A-Za-z0-9.()\/-]{1,})*(?!\S)"
            values_re = r"[-+]?\d+\.\d+e[+-]?\d+"
            names = [re.findall(names_re, name) for name in names]
            names = [name for group in names for name in group]
            names_split = []
            for name in names:
                if len(name) <= max_char_name:
                    names_split.append(name.strip())
                else:
                    remainder_len = len(name) % max_char_name
                    divisor = int(len(name) / max_char_name)
                    names_split.append(name[:remainder_len - divisor].strip())
                    for i in range(int(len(name) / max_char_name)):
                        start = remainder_len - divisor + (i + 1) + i * max_char_name
                        end = start + max_char_name
                        names_split.append(name[start:end].strip())

            values = [re.findall(values_re, val) for val in values]
            values = [val for group in values for val in group]

            if len(names_split) != len(values):
                self.logger.error('Error matching energy values.')

            energies = {}
            for key, val in zip(names_split, values):
                if key == 'Temperature':
                    energies[key] = float(val) * ureg.kelvin
                elif key.startswith('Pres'):
                    key = key.rstrip(' (bar)')
                    energies[key] = float(val) * ureg.bar
                else:
                    energies[key] = float(val) / MOL * ureg.kJ

            return energies

Note: I also had to change the step_info regex slightly, because it was falsely matching another instance of step in my example log file. New code:
            Quantity(
                'step_info',
                r'[\r\n]\s*(Step.+\n[\d\.\- ]+)',
                str_operation=str_to_step_info, convert=False)]

I am now continuing the debugging process...

Ok, this was actually more complicated then we initially thought. The difficulty is dealing with single spaces between keys when the following key reaches the maximum character limit (14). For the record, here is the new code which addresses the issue:
        def str_to_energies(energies_block):

            max_char_name = 14
            rows = energies_block.strip().split("\n")
            names = [rows[i_row] for i_row in range(len(rows)) if i_row % 2 == 0]
            values = [rows[i_row] for i_row in range(len(rows)) if i_row % 2 != 0]

            names_re = r"(?<!\S)[A-Za-z0-9.()\/-]{1,}(?: [\S][A-Za-z0-9.()\/-]{1,})*(?!\S)"
            values_re = r"[-+]?\d+\.\d+e[+-]?\d+"
            names = [re.findall(names_re, name) for name in names]
            names = [name for group in names for name in group]
            names_split = []
            for name in names:
                if len(name) <= max_char_name:
                    names_split.append(name.strip())
                else:
                    remainder_len = len(name) % max_char_name
                    divisor = int(len(name) / max_char_name)
                    names_split.append(name[:remainder_len - divisor].strip())
                    for i in range(int(len(name) / max_char_name)):
                        start = remainder_len - divisor + (i + 1) + i * max_char_name
                        end = start + max_char_name
                        names_split.append(name[start:end].strip())

            values = [re.findall(values_re, val) for val in values]
            values = [val for group in values for val in group]

            if len(names_split) != len(values):
                self.logger.error('Error matching energy values.')

            energies = {}
            for key, val in zip(names_split, values):
                if key == 'Temperature':
                    energies[key] = float(val) * ureg.kelvin
                elif key.startswith('Pres'):
                    key = key.rstrip(' (bar)')
                    energies[key] = float(val) * ureg.bar
                else:
                    energies[key] = float(val) / MOL * ureg.kJ

            return energies

ok I will have a look when pr is ready.

@ladinesa You can continue to ignore these for now, I just want to document the issues I experience.
Next problem:
Something is wrong with the trr file provided, even though it is present. We used to only check xtc file if trr file was missing. So, now I check to see if the positions are present in the MDA universe made from the trr file, and if not, we try the xtc file if it exists. This is in parse() function:
        # I have no idea if output trajectory file can be specified in input
        trr_file = self.get_gromacs_file('trr')
        trr_file_nopath = trr_file.rsplit('.', 1)[0]
        trr_file_nopath = trr_file_nopath.rsplit('/')[-1]
        xtc_file = self.get_gromacs_file('xtc')
        xtc_file_nopath = xtc_file.rsplit('.', 1)[0]
        xtc_file_nopath = xtc_file_nopath.rsplit('/')[-1]
        if not trr_file_nopath.startswith(self._basename):
            trajectory_file = xtc_file if xtc_file_nopath.startswith(self._basename) else trr_file
        else:
            trajectory_file = trr_file

        self.traj_parser.mainfile = topology_file
        self.traj_parser.auxilliary_files = [trajectory_file]
        # check to see if the trr file can be read properly (and has positions), otherwise try xtc file instead
        positions = getattr(self.traj_parser, 'universe', None)
        positions = getattr(positions, 'atoms', None) if positions else None
        positions = getattr(positions, 'positions', None) if positions else None
        if positions is None:
            self.traj_parser.auxilliary_files = [xtc_file] if xtc_file else [trr_file]

A few more things that I corrected so far:

Another issue with the energy regex. Sometimes there is no empty line after the energies block, some error in writing to the file I guess. Instead there is a line about the load imbalance. So, I changed the regex of grabbing the energy block to r'Energies \(kJ/mol\)\s*([\s\S]+?)\nD.* step.* load imb.*|\n\n',. Not sure how generalizable this is, but we can see if other cases occur.

There was some error occurring in the storage of the Rg values within the archive. This probably had to do with varying lengths of the calculation and system lists. I re-coded this function within the workflow normalizer, so that it now properly deals with the system references within calculation.

While addressing #2 above, I realized that the archive_to_universe function in atom_utils did not properly set the trajectory times. I now added some code to import the spacing between system frames (in terms of physical times) within this function.

The above fixing resulted in the first example file provided by the user to be parsed properly. I am now continuing to test the remaining examples...

@ladinesa Could you take a look at some point at the loop over atoms (for n in range(n_atoms):) in the parse_method() function of the gromacs parser?
The data that the user supplied is a relatively large system (~150k atoms). I put some restrictions in place in terms of calculating rdfs and msds, so that this is skipped for molecule types that have more than some cutoff number of molecules, since in many cases for protein simulations there are tens of thousands of water molecules, and calculating these quantities is a waste. (I guess in general it is questionable how useful these things are anyway, this is something we will assess in the future as custom observables are supported).
In any case, once I made these restrictions, I timed the various functions to see where the bottleneck of processing was:
done with initial parsing: 0.036649219195048016 min
done with method parsing: 3.9186079184214275 min
done with atoms loop in parse_method: 3.5282757679621377 min
done with system parsing: 0.2758171478907267 min
done with calculation parsing: 0.049816056092580156 min
done with input parsing: 4.337231318155924e-05 min
done with workflow parsing: 1.7323389212290445 min
done with calculating rdfs: 0.05297237237294515 min
done with calculating msds: 1.6793344179789225 min
A decent portion of the parsing time is taken in this atoms loop in parse_method, ~3.5 min out of ~7 min for complete processing (parsing + normalization). I was just wondering if we could make this more efficient, especially since parse_system() is also dealing with the atoms list and creates the entire topology but only takes ~0.3 min.

@ladinesa Could you take a look at some point at the loop over atoms (for n in range(n_atoms):) in the parse_method() function of the gromacs parser?
The data that the user supplied is a relatively large system (~150k atoms). I put some restrictions in place in terms of calculating rdfs and msds, so that this is skipped for molecule types that have more than some cutoff number of molecules, since in many cases for protein simulations there are tens of thousands of water molecules, and calculating these quantities is a waste. (I guess in general it is questionable how useful these things are anyway, this is something we will assess in the future as custom observables are supported).
In any case, once I made these restrictions, I timed the various functions to see where the bottleneck of processing was: done with initial parsing: 0.036649219195048016 min done with method parsing: 3.9186079184214275 min done with atoms loop in parse_method: 3.5282757679621377 min done with system parsing: 0.2758171478907267 min done with calculation parsing: 0.049816056092580156 min done with input parsing: 4.337231318155924e-05 min done with workflow parsing: 1.7323389212290445 min done with calculating rdfs: 0.05297237237294515 min done with calculating msds: 1.6793344179789225 min
A decent portion of the parsing time is taken in this atoms loop in parse_method, ~3.5 min out of ~7 min for complete processing (parsing + normalization). I was just wondering if we could make this more efficient, especially since parse_system() is also dealing with the atoms list and creates the entire topology but only takes ~0.3 min.

can you send me the file so I can benchmark it reliably? Have to tried commenting out the loop to see if this is really the bottle neck?

Yes, if you comment out the loop, the entire parsing takes 3.5 min less.
I have included 2 tests in HU box, since the zip is ~1.5GB: https://box.hu-berlin.de/f/fbde8aa6b21c4f6e9140/?dl=1
There are 2 test systems there, main files = npt2.log, umbrella2.log. Each have ~150k atoms, but umbrella2.log has a longer trajectory.
NOTE: For testing these files you should either:

use the current version of this branch (I just pushed my latest changes which also include some simple print statements with the timings of each part of the code) and also the corresponding version of NOMAD - https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1643 -- These include my changes which avoid calculating the rdfs and msds for the >100k water molecules. Otherwise the parsing will take forever.
If you want to just use the dev version, simply comment out parse_workflow() and only parse (i.e., don't run the normalization)

Let me know if you need anything else or want to discuss. I want to make a few more changes to the code, and then I will probably be ready for the PR

Yes, if you comment out the loop, the entire parsing takes 3.5 min less.
I have included 2 tests in HU box, since the zip is ~1.5GB: https://box.hu-berlin.de/f/fbde8aa6b21c4f6e9140/?dl=1 There are 2 test systems there, main files = npt2.log, umbrella2.log. Each have ~150k atoms, but umbrella2.log has a longer trajectory.
NOTE: For testing these files you should either:

use the current version of this branch (I just pushed my latest changes which also include some simple print statements with the timings of each part of the code) and also the corresponding version of NOMAD - https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1643 -- These include my changes which avoid calculating the rdfs and msds for the >100k water molecules. Otherwise the parsing will take forever.
If you want to just use the dev version, simply comment out parse_workflow() and only parse (i.e., don't run the normalization)

Let me know if you need anything else or want to discuss. I want to make a few more changes to the code, and then I will probably be ready for the PR

I cannot think of any way to address this rather than creating AtomParameters only for unique species. Is this okay?

Yes, if you comment out the loop, the entire parsing takes 3.5 min less.
I have included 2 tests in HU box, since the zip is ~1.5GB: https://box.hu-berlin.de/f/fbde8aa6b21c4f6e9140/?dl=1 There are 2 test systems there, main files = npt2.log, umbrella2.log. Each have ~150k atoms, but umbrella2.log has a longer trajectory.
NOTE: For testing these files you should either:

use the current version of this branch (I just pushed my latest changes which also include some simple print statements with the timings of each part of the code) and also the corresponding version of NOMAD - https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1643 -- These include my changes which avoid calculating the rdfs and msds for the >100k water molecules. Otherwise the parsing will take forever.
If you want to just use the dev version, simply comment out parse_workflow() and only parse (i.e., don't run the normalization)

Let me know if you need anything else or want to discuss. I want to make a few more changes to the code, and then I will probably be ready for the PR

I cannot think of any way to address this rather than creating AtomParameters only for unique species. Is this okay?

Hmmm, I think it would be ok for many cases. Typically mass and charge is set via atomtypes in gromacs, so that would be consistent, however it is possible to overwrite these for individual atoms. Can we default to doing it by species type (and by this I mean the label within atom parameters not the atomic element, the labels are more specific generally), and then do some sort of check for inconsistencies? Or will this slow your approach down anyway?

default

do some sort of check for inconsistencies

to check for inconsistency you mean to make sure charge, resid, motype etc is the same for the same label? If not, I should also write it as an AtomParameter?

AtomParameters

yes this is what i meant. I will create a new branch to implement this from develop or from your working branch?

In my example, charges vary for the same label. should I include it in the comparison?

In my example, charges vary for the same label. should I include it in the comparison?

I think if any of the attributes differ then we need to create a new group, unless you have some other way to keep tracking of the varying attributes. It may end up being too tedious or defaulting to the old way. One thing that may help this is to improve the storage of the FF, which will define these things more clearly...but that is out of scope for now.

have a look at the changes in a9e769c79734fb7081602a44dc62d230e5b341f3

have a look at the changes in a9e769c

Yeah this looks good to me! Have you considered downstream effects though? I know, e.g., that in the archive_to_universe function in atomutils we use the masses, etc., to build the universe. This is then used to calculate COM for the rdf and msd calculation. We need to check any usage of the atom parameters throughout the code.

@ladinesa FYI - I had to revert the MDAnalysis requirement to version 2.1.0. Version 2.5.0 has some errors when reading some versions of trr files. All the tests within the atomistic parsers pass, but just wanted to let you know in case you had some other usage somewhere that needed the newer version (I presume not and I am the one who put 2.5.0, but I can't remember).
On a slightly related note. I always get this warning:
WARNING  nomad.client         2023-08-03T07:28:33 Incompatible version of MDAnalysis.
  - nomad.client.parser: parsers/gromacs
  - nomad.commit:
  - nomad.deployment: devel
  - nomad.service: unknown nomad service
  - nomad.version: 1.2.1.dev45+g1514d2286.d20230803

Even when I am using the version of MDA specified in pyproject.toml. Do you know where this is coming from? I would like to correct this.

@ladinesa FYI - I had to revert the MDAnalysis requirement to version 2.1.0. Version 2.5.0 has some errors when reading some versions of trr files. All the tests within the atomistic parsers pass, but just wanted to let you know in case you had some other usage somewhere that needed the newer version (I presume not and I am the one who put 2.5.0, but I can't remember).
On a slightly related note. I always get this warning:
WARNING  nomad.client         2023-08-03T07:28:33 Incompatible version of MDAnalysis.
  - nomad.client.parser: parsers/gromacs
  - nomad.commit:
  - nomad.deployment: devel
  - nomad.service: unknown nomad service
  - nomad.version: 1.2.1.dev45+g1514d2286.d20230803

Even when I am using the version of MDA specified in pyproject.toml. Do you know where this is coming from? I would like to correct this.

This is coming from get_force_field_parameters, where I used the reader from version 2.0.0. I am really happy about this function. Maybe we can remove this or not use it for now until we find a more robust solution.

have a look at the changes in a9e769c

Yeah this looks good to me! Have you considered downstream effects though? I know, e.g., that in the archive_to_universe function in atomutils we use the masses, etc., to build the universe. This is then used to calculate COM for the rdf and msd calculation. We need to check any usage of the atom parameters throughout the code.

Actually, I did have a warning about the atomtypes not having a length not equal to number of atoms during normalization. Not sure though how to deal with it without digging a bit.

have a look at the changes in a9e769c

Yeah this looks good to me! Have you considered downstream effects though? I know, e.g., that in the archive_to_universe function in atomutils we use the masses, etc., to build the universe. This is then used to calculate COM for the rdf and msd calculation. We need to check any usage of the atom parameters throughout the code.

Actually, I did have a warning about the atomtypes not having a length not equal to number of atoms during normalization. Not sure though how to deal with it without digging a bit.

I can try to go through and address this if you would like. I have just been digging through the Gromacs parser, so I may be able to find these things quickly.
I guess the question is how we deal with this more generally. For example, are we going to make equivalent changes for the Lammps parser as well? How will a user know whether the AtomParameters are grouped or not in general?

rameters are

It seems that we cannot sample AtomParameters like frames right? In that case, we cannot do anything about it, if it takes an additional 5 mins to write 150k entries then so be it.

Let me think about this more once I am done with this other PR...


Question about internal & external IPs.

I'm quite new to docker and weblogic, so I don't really know where to comment the problem I'm having, so I'll do it here.
Okay, let's start by the environment.
the project I'm wotking at requires weblogic 12.1.3 as well as coherence 12.1.3
So I create an AdminServer and some machines for "data" and "proxy".
If I test and try to connect with the proxy docker machine from inside the "host" computer everything works flawlessly.
But when I try to connect from an outside machine (that obiously doesn't have acces to the 172.x.x.x internal docker IPS) the connection fails.
In the first place it seems to work but In a second attempt the external machine tries to connect to an internal IP, and then fails.
I've created the proxy node with this configuration
docker run -d -v /webdav:/mnt/webdav_host -v /deploy:/mnt/deploy -v /logs:/mnt/logs_host -t -e MS_HOST=serverExternalIP -e NM_NAME=Machine-proxy1 -e MS_NAME=proxyServer-1 --lin
k wlsadmin:wlsadmin -p 7010:7001 -p 9099:9099 --name=proxy_1 weblogic/sample:12.1.3 createServer.sh
And the trace I get from the application running at the outside machine is something like this:
Connecting to ServerExternalIP:9099 --- OK
Connecting to ServerExternalIP:9099.3 --- OK
Connecting to 172.x.x.x:9099 -- Fail
There is some way to configure docker or Weblogic to show the server public IP instead of the docker machine one?
Thanks in advance.

It is a networking issue with the way Docker works. You should review the Docker Networking section in the documentation to get a better understanding.
https://docs.docker.com/engine/userguide/networking/


flyway-core depends on jackson: fails if not present

Which version and edition of Flyway are you using?
Version 9.19.2
If this is not the latest version, can you reproduce the issue with the latest one as well? (Many bugs are fixed in newer releases and upgrading will often resolve the issue)
yes
Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)
flyway-core java API
Which database are you using? (Type & version)
postgresql
Which operating system are you using?
linux
What did you do? (Please include the content causing the issue, any relevant configuration settings, the SQL statement(s) that failed (if any), and the command you ran)
upgraded flyway-core from 9.19.1 to 9.19.4
this excerpt reproduces the problem
final Flyway flyway = Flyway
                .configure()
                .configuration(Map.of("flyway.postgresql.transactional.lock", "false"))
                .load();

What did you expect to see?
Expected flyway-core to work normally, even if jackson is not present
What did you see instead?
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
	at org.flywaydb.core.api.configuration.ClassicConfiguration.determineKeysToRemoveAndRemoveFromProps(ClassicConfiguration.java:1910)
	at org.flywaydb.core.api.configuration.ClassicConfiguration.configure(ClassicConfiguration.java:1479)
	at org.flywaydb.core.api.configuration.FluentConfiguration.configuration(FluentConfiguration.java:959)
	at org.example.Main.main(Main.java:11)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 4 more

This seems similar to https://github.com/flyway/flyway/issues/3672. jackson is marked as optional on the pom.xml. Also why is two different json libraries being used ?

And the two different json libraries being used thing will also be taken care of.


error: attribute missing

When I tried to review a PR using nix-shell -p nixpkgs-review --run "nixpkgs-review pr 169058" I received an error about a missing attribute:
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs release-21.11:refs/nixpkgs-review/0 pull/169058/head:refs/nixpkgs-review/1
$ git worktree add /home/henri/.cache/nixpkgs-review/pr-169058-1/nixpkgs c093b038d8df8bfcec81d62219b093440adf3e53
Preparing worktree (detached HEAD c093b038d8d)
Updating files: 100% (29052/29052), done.
HEAD is now at c093b038d8d Merge pull request #168903 from NixOS/backport-168855-to-release-21.11
$ git merge --no-commit --no-ff 6bee0550531bf20a7e291393ded04402c3808b30
Automatic merge went well; stopped before committing as requested
$ nix --experimental-features nix-command build --no-link --keep-going --option build-use-sandbox relaxed -f /home/henri/.cache/nixpkgs-review/pr-169058-1/build.nix
error: attribute 'bijiben' missing

       at /home/henri/.cache/nixpkgs-review/pr-169058-1/build.nix:71:9:

           70|         gnome-recipes
           71|         gnome."bijiben"
             |         ^
           72|         gnome."cheese"
(use '--show-trace' to show detailed location information)

What is weird, however, is that when I take the command before the error and paste it into the spawned shell it runs through without problems.
nixpkgs-review version: https://github.com/NixOS/nixpkgs/blob/faad370edcb37162401be50d45526f52bb16a713/pkgs/tools/package-management/nixpkgs-review/default.nix

That is a bug in ofborg or a stale ofborg result. Try nix-shell -p nixpkgs-review --run "nixpkgs-review pr --eval local 169058"

This might also resolve it: rm -r /home/henri/.cache/nixpkgs-review

Thank you, this works really well! Would it be possible to print a more helpful error message in these cases or automatically attempt a local evaluation?

Yeah, we should display a better error message but I am against trying another eval because it uses a lot of RAM.


[Plain Language] Add back ReportList; Remove Accordion; Add slash checker

Hi @sofisa16 , since it is just one line code, I add back ReportList for better checking the report result from Slash checker, and complex word. Thank you.


Product Create Page- Public Cloud

Create the product create page for a public cloud project.
Note that a lot of the components from the private cloud page can be re used, but also custom components will need to be created. Note that we can and should re use the validation logic from the current UI where possible.
Private Zenhub Image
Dod:
The UI for the create page is created and is connected to the API endpoint

Unable to test because for now route leads to 404. Looks like the same bug blocking #1099.

edited rotes for create button, waiting for review

Now it's blocked by backend failure. Searching for contact doesn't work:

Now everything works fine.
Found a small typo - no spaces around the link:


Dashboard examples are run using Rust 2018 edition

Dashboard examples are run using Rust 2018 edition but they should use Rust 2021. This causes preprocessing warnings or failures in some examples where backwards compatibility has not been kept. For example, in Rust by Example/Functions/Closures/Examples in std/Searching through iterators/22.rs we get this:
stderr:

warning: this method call resolves to `<&[T; N] as IntoIterator>::into_iter` (due to backwards compatibility), but will resolve to <[T; N] as IntoIterator>::into_iter in Rust 2021
--> src/test/dashboard/books/Rust by Example/Functions/Closures/Examples in std/Searching through iterators/22.rs:26:47
|
26 |     println!("Find 2 in array2: {:?}", array2.into_iter().find(|&&x| x == 2));
|                                               ^^^^^^^^^
|
= note: `#[warn(array_into_iter)]` on by default
= warning: this changes meaning in Rust 2021
= note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/IntoIterator-for-arrays.html>
help: use `.iter()` instead of `.into_iter()` to avoid ambiguity
|
26 |     println!("Find 2 in array2: {:?}", array2.iter().find(|&&x| x == 2));
|                                               ~~~~
help: or use `IntoIterator::into_iter(..)` instead of `.into_iter()` to explicitly iterate by value
|
26 |     println!("Find 2 in array2: {:?}", IntoIterator::into_iter(array2).find(|&&x| x == 2));
|                                        ++++++++++++++++++++++++      ~

Looks like edition processing in the dashboard code is not working well for 2015 examples. There are currently 4 examples which are expected to use Rust 2015 but nothing gets added. For example:
"rmc-rustc" "-Z" "no-codegen" "src/test/dashboard/books/The Rust Reference/Names/Paths/183.rs"

While examples that were using the 2018 edition are called as:
"rmc-rustc" "--edition" "2018" "-Z" "no-codegen" "src/test/dashboard/books/Rust by Example/Flow of Control/for and range/32.rs"


Determine if & which MSBuild has been previously defined

Hello, is there a way to determine if "use 15.0 MSBuild" has been defined previously? I have some build files that need to use a specific version of msbuild, but in my "common code" i have a default value set.
So is there a way to do something like this in my common code:
if (!$msbuild) {
  use 15.0 MSBuild
}

Thanks in advance!

This may be done in several ways. I would use Resolve-MSBuild instead of use (use is not deprecated but kind of legacy).
The shortest way. In the build script scope, do:
Set-Alias MSBuild (Resolve-MSBuild *)
Set-Alias MSBuild15 (Resolve-MSBuild 15.0)

Then use MSBuild15 instead of MSBuild where exactly v15 is needed.
For version checks, having set Set-Alias MSBuild (Resolve-MSBuild *), you can get its version as
(Get-Item (Get-Alias MSBuild).Definition).VersionInfo.FileVersionRaw.Major

Alternatively, you can just use use 15.0 MSBuild right in a task which requires v15.
The alias is created locally for this task only, the parent alias, if any, stays.
# Latest
use * MSBuild

# Use 15.0
task build15 {
    use 15.0 MSBuild
    Write-Build Yellow (MSBuild /version /nologo)
}

# Use Latest
task buildLatest {
    Write-Build Yellow (MSBuild /version /nologo)
}

task . build15, buildLatest

And finally here is the version with variables
# latest
$MSBuild = Resolve-MSBuild

# define if not yet
if (!$MSBuild) {
	$MSBuild = Resolve-MSBuild
}

# use latest
Write-Host (& $MSBuild /version /nologo)

# redefine with specific version
if (!$MSBuild -or ((Get-Item $MSBuild).VersionInfo.FileVersionRaw.Major -ne 15)) {
	$MSBuild = Resolve-MSBuild 15.0
}

# use 15.0
Write-Host (& $MSBuild /version /nologo)

@nightroman That is perfect,
(Get-Item (Get-Alias MSBuild).Definition).VersionInfo.FileVersionRaw.Major
Is just what I need.
Thank you!


Few questions

Hi,
Not sure if the right place, but:

what would be the difference of secor vs a Kafka Connect to S3?
can secor be used to recover a kafka cluster in case of disaster? Meaning rebuilding a Kafka Cluster entirely from a secor backup

Thanks,
Stephane

secor vs. kafka connect, secor is more lightweight and yet still provides capability to parse the message in different format: Thrift/JSON and generate output in compressed/uncompressed, SequenceFile/Parquet format;
For restoring, there is a OffsetMessageParser in secor which keep the kafka messages in its original format for restoring purpose.

Hi @HenryCaiHaiying thanks a lot for your answer. Regarding the restoring, is there something that exists out there that demonstrates how to restore messages using the OffsetMessageParser?
Also, is there something like a consumer that would consume Kafka data from S3, or do we have to create our own?

There is no ready-made restore tool, but you can write a tool to publish
back those messages back to kafka since the messages are saved in the exact
offset order corresponding to their kafka position.
The consumer is already included in secor code, there is no need to create
another one.
On Tue, Dec 27, 2016 at 2:00 PM, Stephane Maarek<EMAIL_ADDRESS>wrote:

Hi @HenryCaiHaiying https://github.com/HenryCaiHaiying thanks a lot for
your answer. Regarding the restoring, is there something that exists out
there that demonstrates how to restore messages using the
OffsetMessageParser?
Also, is there something like a consumer that would consume Kafka data
from S3, or do we have to create our own?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/296#issuecomment-269388675,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHCX_wfi9OziMdSZoRya_PaVi5I0ZWO2ks5rMYpqgaJpZM4LSmRF
.

Thanks! By consumer, I meant consumer that reads from S3, not the Secor Consumer that reads from Kafka. Basically the use case is that someone can "replay" data using a special consumer directly from S3. Is there such thing?

We don't have ready-to-made code for that, but I don't think it will be
hard to write some code to read the file from S3 and then publish the
record back to Kafka.
On Tue, Dec 27, 2016 at 7:13 PM, Stephane Maarek<EMAIL_ADDRESS>wrote:

Thanks! By consumer, I meant consumer that reads from S3, not the Secor
Consumer that reads from Kafka. Basically the use case is that someone can
"replay" data using a special consumer directly from S3. Is there such
thing?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/296#issuecomment-269418817,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHCX_6NfThp7jb_Y8Wqj1N2erObyqgpUks5rMdPdgaJpZM4LSmRF
.

Thanks!
Finally, are there plans to support the timeindex of kafka <IP_ADDRESS> soon? I don't believe those would be lost with the current OffsetMessageParser class?

For timeindex, do you mean rewind to a particular offset point
corresponding to a timestamp?  It might be just easier to write a separate
tool to rewind the secor consumer's offset point to those points.  The
capability of getting the offset point for a particular time existed in
kafka 8 as well, it's just not as accurate, it can rewind to an offset
which is a little before the time you gave.
On Wed, Dec 28, 2016 at 9:25 PM, Stephane Maarek<EMAIL_ADDRESS>wrote:

Thanks!
Finally, are there plans to support the timeindex of kafka <IP_ADDRESS> soon?
I don't believe those would be lost with the current OffsetMessageParser
class?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/296#issuecomment-269583552,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHCX_yBNGm4T7pUTTgxvxATwYAd9mM1cks5rM0RFgaJpZM4LSmRF
.

I mean backing up the timeindex that Kafka 0.10.1 writes (it's a new feature of kafka). See offsetsForTimes here: https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

Hi,
Do you have code snippet to implement kafka messages recovery using secor
Regards,
ashwini


Update Blink Provisioning Tutorial

Clarify that currently only users are provisioned in Blink (leaving in references to group member sync via assignment)
Update Blink support request form URL

@mattridley : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@jeevansd: I'm sorry - only the author of this article, @zchia, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

@zchia Please review this PR. Looks good to me but you can decide.

@zchia @jeevansd Can someone please progress this? It has now been open for 2 months.

@zchia Could you review this pull request? Thank you.

#sign-off

@jeevansd: I'm sorry - only the author of this article, @zchia, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


Switch readiness and liveness probes

What this PR does / why we need it:
Apparently readiness and liveness probes were mistaken. It probably
makes more sense for the liveness probe to have an initial delay of 60
seconds. Also 10s is the default for periodSeconds.
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

/ok-to-test
/assign

/lgtm


Prototype: share view model to fix issue

IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).
Fixes #[issue number]
Description
Clear and concise code change description.
Alternative(s) considered
Have you considered any alternatives? And if so, why have you chosen the approach in this PR?
Type
Choose one: (Bug fix | Feature | Documentation | Testing | Code health | Builds | Releases | Other)
Screenshots (if applicable)
Checklist

[ ] I have read and acknowledged the Code of conduct
[ ] I have read How to Contribute
[ ] I have read the Developer's guide
[ ] I have signed the Google Individual CLA, or I am covered by my company's Corporate CLA
[ ] I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach
[ ] I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the style guide of this project
[ ] I have run ./gradlew check and ./gradlew connectedCheck to test my changes locally
[ ] I have built and run the reference app(s) to verify my change fixes the issue and/or does not break the reference app(s)

@jingtang10 This fix works for me

i'll comment on the main issue


Appstudio update test-component-pac-legv

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-legv-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
17 seconds
clone-repository

✅ Succeeded
2 minutes
build-container

✅ Succeeded
30 seconds
deprecated-base-image-check

✅ Succeeded
22 seconds
inspect-image

✅ Succeeded
34 seconds
clamav-scan

✅ Succeeded
1 minute
clair-scan

✅ Succeeded
20 seconds
sbom-json-check

✅ Succeeded
22 seconds
label-check

✅ Succeeded
22 seconds
optional-label-check

✅ Succeeded
4 seconds
show-sbom

✅ Succeeded
5 seconds
show-summary


Fix: Update LinkedUp to work with Motoko base library

Overview
This PR intends to update LinkedUp to work with<EMAIL_ADDRESS>Changes

[x] Standard library is now imported via mo:base/[module]
[x] Standard library modules are now in PascalCase
[x] PrincipalId -> Principal 😄
[x] Bumped dfx version in dfx.json

Hey @chenyan-dfinity , can we use this PR for the update so we can merge your GitPod PR?


After the error is encountered, trinity does not retry the data chunk (i.e. doesn't recover to normal operation), but also does not stop its processes.
DEBUG output in gist.
First time I see this.
How can it be fixed
Haven't investigated.

Yikes, so I have no idea what to do if leveldb is corrupted. I think we just have to tell the user to dump the whole db and try again.
As for things continuing to run, it looks like fast sync ended, and we have a weird thing where fast sync ending just rolls over into regular sync. I think we've talked about having a service exit code (since the caller who started the service can't currently tell if it ended cleanly), but I can't find an issue for that. So this issue can serve as a reminder.

Had this happen on regular sequential full-sync, too, so edited title.

Telling the user to dump DB and start from scratch isn't very helpful, I'm afraid. This forces the node operator to either make manual DB backups (somewhat wasteful) or run several nodes behind a load-balancer (wasteful for many use cases and operationally complex).
The issue would "solve itself" (in a sense) if historic snapshots are available (during full-sync; not really considering fast-sync anymore); a roll-back could then be performed, theoretically.

Given that this is a leveldb database issue I'm not quite sure how to come at this.  ideally we need a way to reliably reproduce this error so we can know why it happens.  Any chance you've recently updated your version of plyvel or libsnappy or your system leveldb?  can you pull those versions as well for us?

I think the corruption in both cases happened due to system lock-up on restore-from-suspend, with trinity not safely terminated.
Surely was the case the first time around; not sure about the later one.

System packages:

snappy is v1.1.7 and hasn't been updated since 2017-12-02;
leveldb is v1.21 and hasn't been updated since 2019-04-04 (before this issue was opened);
python-plyvel not installed system-wide.

Virtualenv Python packages python-snappy and plyvel are same as in OP-linked gist and haven't changed since then.

:+1: hoping to address the dirty shutdown issue with our conversion to curio and dropping the multiprocess-based object sharing for our database in the near term future.

Telling the user to dump DB and start from scratch isn't very helpful, I'm afraid. This forces the node operator to either make manual DB backups (somewhat wasteful) or run several nodes behind a load-balancer (wasteful for many use cases and operationally complex).

Oh yeah, for sure it's a terrible thing to tell operators. I'm just not sure it's something that makes sense to fix

The issue would "solve itself" (in a sense) if historic snapshots are available (during full-sync; not really considering fast-sync anymore, in light of issue #511); a roll-back could then be performed, theoretically. Possibly relevant py-evm issue: ethereum/py-evm#1759

I'm not even sure that leveldb snapshots will work if it's corrupted that badly
https://github.com/wbolster/plyvel/issues/8#issuecomment-17480039
Maybe something busted in snappy? ^


Changing min/max zoom when editing style does not work correctly

After editing both values seem to be the same.

Fixed by https://github.com/kalisio/kdk/commit/62929495d1e17ae0f09a8b0dea0b4c7ca54c0a60.


Compilation failed

I get an error when accessing any /api path, but the api return a http code 200:
curl --request GET \
  --url https://{{base_url}}/api/collections \
  --header 'authorization: Bearer {{token}}'

Warning: preg_match(): Compilation failed: quantifier does not follow a repeatable item at offset 0 in /var/www/html/modules/Cockpit/rest-api.php on line 77

I use the latest version (as today) of the docker image agentejo/cockpit. I also created one collection and one entry.

did you define any rules for the api key?

Yes (For testing):
*
/api/collections/get
/api/forms/post/contact

I think having both * and some path define cause the compilation error. Removing * or the paths fix the compilation issue!

* doesn't make any sense, as this would allow any route


Invalid response from node

I'm trying to use smidgen multisig tool from bitfinex and when using carriota field as the node provider, it is not able to handle the response. When using other nodes like https://nodes.thetangle.org:443 everything is fine.
$ smidgen get-balance "XFAELMMFKBKNGAYLKNFOLHMOTAM9IVOLABDLNKNWUIFK9QJXEJKGAUHMEPMYPGLPFBWZAXJMFZYCWMQEAFMGNQKVFZ" --provider "https://field.carriota.com:443"
ERR! Received invalid response from node.
ERR! Retry command or try to use another node.
ERR!
ERR! Use --provider to switch node.

$ smidgen get-balance "XFAELMMFKBKNGAYLKNFOLHMOTAM9IVOLABDLNKNWUIFK9QJXEJKGAUHMEPMYPGLPFBWZAXJMFZYCWMQEAFMGNQKVFZ" --provider "https://nodes.thetangle.org:443"
Balance:<PHONE_NUMBER>54300 (218884849.0543 Mi)

Is it possible that your response it not standard compliant?
Or maybe it is a bug on smidgen's side, although I read others having similar issues with different tools...

Another example is the isf-jclient from iotaspam.com
[2018-02-28 09:33:16] [NodeMngr] [DBG] node 'https://field.carriota.com:443' [0] is not synced (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target), changing node to 'http://<IP_ADDRESS>:14265'

Afaik the golang module also does it, but its not a bug, but a feature ;)
The Cert chain is missing, see: https://www.ssllabs.com/ssltest/analyze.html?d=field.carriota.com&s=<IP_ADDRESS>
Or verify yourself:
openssl s_client -showcerts -connect field.carriota.com:443

This should be fixed now. Added the correct SLL certificates chain to the load balancer.

Thank you, works now!


Enhanced accepting store data sanitation

Fixes #392 (partly)
Ansatz:
Suche nach der Postleitzahl der AS¹ und Abgleich der BB des Ergebnisses mit Koordinaten der AS

Fall 1: Kein Ergebnis/Keine PLZ:
-> Suche (Geocoding) nach Adresse² und Verwendung der PLZ des ersten nahegelegenen³ Ergebnisses
Fall 2: Koordinaten der AS außerhalb der BB der PLZ:
-> Suche (Geocoding) nach Adresse²

Fall 2a: Ergebnis mit gleicher PLZ:
-> Verwendung der Koordinaten des Ergebnisses
Fall 2b: Nahegelegenes³ Ergebnis:
-> Verwendung der PLZ des Ergebnisses

Fall 3: Koordinaten der AS innerhalb der BB der PLZ:
-> Keine Änderung der Daten, scheint valide

¹: Alle PLZ sind im Geocoding Backend mit BB hinterlegt. Ausnahme: PLZ von Postfächern -> Keine Überprüfung der Daten.
²: Suche nach Adresse: Zuerst Suche mit Stadt + Straße + Hausnummer, falls keine Ergebnisse Suche mit Stadt + Straße, falls keine Ergebnisse Suche mit Stadt.
³: Metrik "nahegelegen": BB des Ergebnisses enthält Koordinaten ODER Koordinaten des Ergebnisses weniger als X Kilometer entfernt (X = 5km). Entfernung insbesondere bei Ergebnissen mit Straße ohne Hausnummer nötig, da BB oft nur Fläche der Straße und nicht angrenzende Häuser enthält.
AS: Akzeptanzstelle
PLZ: Postleitzahl
BB: Bounding Box

This is only a first iteration. This identifies 35 stores with wrong postal code/coordinates (out of ~5500).

28 are correctly sanitized (better coordinates/correct postal code)
6 are without nominatim match and remain unchanged (because they are outside of Bavaria or have a typo in the street name)
1 is wrongly changed (PLZ of Rathausplatz, Freyung is changed to that of Rathausplatz, Waldkirchen, Freyung as the former is missing in nominatim)

@maxammann @Schedulaar please let me know what you think. Of course the results are not perfect but I think they are pretty good already. If you want we can discuss/I can explain this in a call.

Coordinates of 'Modehaus Stefan Schick, Forchheim, Marktplatz, 6 & 7' was changed from '48.2553511081981, 12.4161624706892' to new '49.7189649, 11.0611276'.
Postal code of 'Kletterpark Prien am Chiemsee, Prien, Harrasser, Straße' was changed from '83109' to new '83209'.
Postal code of 'Kletterpark Waldbad Anif, Anif, Waldbadstraße' was changed from 'null' to new 'null'.
Postal code of 'Markt Peiting - Gemeinde- und Pfarrbücherei, Peiting, Hauptplatz, 2' was changed from '86791' to new '86971'.
Postal code of 'Kletterhalle Birkenfeld, Birkenfeld, Weidenmühle, 1' was changed from '97833' to new '97834'.
Postal code of 'Jugendzentrum MainHaus/Minigolf Projekt, Marktheidenfeld, Lengfurter Straße, 25' was changed from '97827' to new '97828'.
Postal code of 'Kegelbahn / Historischer Keller, Rieneck, Schulgasse, 4' was changed from '97784' to new '97794'.
Coordinates of 'Krug Optik, Burgsinn, Burgweg, 2' was changed from '50.054901, 9.69414' to new '50.1478801, 9.6505954'.
Postal code of 'Marktgemeinde Dietenhofen, Dietenhofen, Pestalozzistraße, 6' was changed from '92599' to new '90599'.
Postal code of 'Gemeinde Bischofswiesen - Naturbad Aschauerweiher, Bischofswiesen, Aschauerweiherstr., 85' was changed from '83486' to new '83483'.
Postal code of 'Gemeinde Bischofswiesen - Tourist-Info, Bischofswiesen, Hauptstraße, 40' was changed from '83486' to new '83483'.
Coordinates of 'Gasthof Odenwald, Schneeberg, Amorbacher Str., 4' was changed from '49.2173, 9.21635' to new '49.6412578, 9.2437805'.
Postal code of 'Hagebaumarkt, Pfaffenhofen, Josef-Fraunhofer-Str., 21' was changed from '86276' to new '86276'.
Postal code of 'Tanzschule Köpke Rupprecht, Nürnberg, Harmoniestr., 8-10' was changed from 'null' to new '90489'.
Coordinates of 'Stadtverwaltung Neunburg vorm Wald, Neunburg vorm Wald, Schrannenplatz, 1' was changed from '48.085609, 16.283588' to new '49.3486385, 12.3858734'.
Coordinates of 'Neunburger Freunde der Musik, Neunburg v. W., Bockskirn, 19' was changed from '45.357616, 1.880586' to new '49.3634526, 12.4014459'.
Coordinates of 'BayWa Neunburg vorm Wald, Neunburg v. W., Bahnhofstraße, 28' was changed from '48.388565, 1.371688' to new '49.3447092, 12.3736702'.
Coordinates of 'Festspielverein Neunburg v. W., Neunburg v. W., Schrannenplatz, 3' was changed from '48.085609, 16.283588' to new '49.3486385, 12.3858734'.
Coordinates of 'Musikschule Neunburg vorm Wald, Neunburg v. W., Im Berg, 12' was changed from '35.95042, 126.656906' to new '49.3490743, 12.38347'.
Postal code of 'KW Fotografie, Großheubach, Röllbacher Str., 72' was changed from '63927' to new '63920'.
Postal code of 'OBI Würzburg Filiale Würzburg-Hafen, Würzburg, Paradiestraße, 5' was changed from '97084' to new '97084'.
Postal code of 'Stadt Freyung, Freyung, Rathausplatz, 1' was changed from '94078' to new '94065'.
Coordinates of 'Haushaltswaren Albrecht, Rain am Lech, Bahnhofstraße, 15' was changed from '47.20829, 10.14177' to new '48.69408, 10.9222539'.
Coordinates of 'Ludwig Erhard Zentrum, Fürth, Friedrichstr., 4' was changed from '51.450672, 7.001397' to new '49.4734927, 10.9912142'.
Coordinates of 'Sparkasse Eichenbühl, Eichenbühl, Hauptstr., 63' was changed from '49.701309, 9.258349' to new '49.701276, 9.333205983496095'.
Postal code of 'Bäumler Gartentechnik, Nürnberg, Eibacher Hauptstr., 62' was changed from '90541' to new '90451'.
Postal code of 'Autoteile Schwer, Vöhringen, An der Alten Ziegelei, 7a' was changed from '89296' to new '89269'.
Coordinates of 'Casa Rossa, Buchhandlung, Miltenberg, Hauptstr., 68' was changed from '51.084396, 13.2902' to new '49.7009888, 9.2569735'.
Postal code of 'Ziptransfer, Spain, C. Llovera 53, 5º 1ª 43201, Reus' was changed from 'null' to new 'null'.
Coordinates of 'Toni Paulis Befestigungstechnik, Tann, Sattlerweg, 1' was changed from '51.410507, 12.302841' to new '48.3227821, 12.8866222'.
Coordinates of 'Jagdrevier Kleinschwarzenlohe, Wendelstein, Kornburger Str., 10a' was changed from '49.403954, 11.112794' to new '49.3494588, 11.1124916'.
Postal code of 'Confiserie Dengel, Erding, Haager Str., 11' was changed from '85443' to new '85435'.
Coordinates of 'CHARME Miederwaren, Amberg, Rathausstr., 4' was changed from '48.72142, 8.360489' to new '49.4450066, 11.8592097'.
Coordinates of 'MOSER Trachten GmbH, Amberg, Fuggerstr., 1+7' was changed from '48.123512, 11.536462' to new '49.4449298, 11.8272354'.
Coordinates of 'hair and fashion Monika Hirsch / NICOWA Mode, Amberg, Windhorststr., 22' was changed from '49.172958, 8.020514' to new '49.4454211, 11.858728'.

Overall, it seems very solid!
I think that Nominatim is not as good as I expected for some of the queries (especially when it comes to house numbers or these small annoying typos), but it's what we have, I guess.
Great work!

Somehow I only reviewed this partially. Going to review it again.

'Kletterpark Waldbad Anif, Anif, Waldbadstraße' was changed from 'null' to 'null'

This message is kind of weird. What does it mean?

'Kletterpark Waldbad Anif, Anif, Waldbadstraße' was changed from 'null' to 'null'

This message is kind of weird. What does it mean?

It means that the postal code was previously null and is still null after geocoding, so it was not possible to find a correct postal code. Will improve the message.

'Kletterpark Waldbad Anif, Anif, Waldbadstraße' was changed from 'null' to 'null'

This message is kind of weird. What does it mean?

It means that the postal code was previously null and is still null after geocoding, so it was not possible to find a correct postal code. Will improve the message.

The thing is that the message is missing the postal code. I think the other messages are clear because they mention the postal code.

'Kletterpark Waldbad Anif, Anif, Waldbadstraße' was changed from 'null' to 'null'

This message is kind of weird. What does it mean?

It means that the postal code was previously null and is still null after geocoding, so it was not possible to find a correct postal code. Will improve the message.

The thing is that the message is missing the postal code. I think the other messages are clear because they mention the postal code.

They are missing the postal code because there is no (valid) postal code set :D

Would be perfect if you could add a changelog here: https://github.com/digitalfabrik/ehrenamtskarte/blob/main/CHANGELOG.md
Just the key changes and a small description is sufficient I think.


Unable to bind JSON to object. Reason: Unrecognized field "message"

Mode : standalone.
Exception : Unable to bind JSON to object. Reason: Unrecognized field "message"
Sample of file used :
{
"request": {
"method": "GET",
"url": "vehicle/getAllModelInfo/FordChina/env/A"
},
"response": {
"status": 200,
    "transformers": ["body-transformer"],
"message": null,
"vehicle": [
    {
        "imgURL": null,
        "key": 1,
        "currency": "¥",
        "vehicleName": "福克斯",
        "startingPrice": 12,
        "fuelEfficiencyFrom": "****"

.....}]
}
Is there a way I can add custom field in standalone.

As per the README, please can you post questions on the mailing list.


Revamp docs as UBI has evolved.

Description
UBI has evolved since it was first released in May 2024. This is an update of the documentation. We have introduced an explicit JS client, so we can streamline the documentation by referencing the deployed document.
Issues Resolved
no specific issue
Version
2.18
Frontend features
n/a
Checklist

[x ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Replaces https://github.com/opensearch-project/documentation-website/pull/8799 which has signoff/rebase issues.

@jzonthemtn could you give a quick pass through on the tech side...  I don't think much has changed!

This documentation PR is ready for review.

I wanted to follow up and see if we could get this merged?

@epugh Thanks for following up! I will review this week and put it through editorial review. Please tag me if you have any questions.


allow template creator to specify expected datatype for a literal field

In template creation/specification, allow template creator to optionally specify an expected datatype for a literal field.
For initial implementation, offer these datatypes:

from https://www.w3.org/TR/rdf11-concepts/#xsd-datatypes

xsd:integer
xsd:dateTime
xsd:dateTimeStamp

Extended Date/Time Format (EDTF) (http://id.loc.gov/datatypes/edtf)

Implementation should be flexible enough that additional datatypes could be added (by developers) in future.
see #2890 for enforcing the datatype when data is entered in template
see # for ability to specify expected input via regex

looks good!

@justinlittman is the documentation here sinopia_editor/src/components/vocabulary/Vocab.js:

which is used by sinopia_editor/src/components/App.jsx

That sounds right. This was originally written by @jermnelson. It is possible that new terms have been created but not added to Vocab.js.

see #2859 (I think this is relevant)

@michelleif seems like #2859 is relevant.  I will take the ticket as it jibes with this work.

hurrah, thank you!


[BUG] useQueryParam is not safe typely, runtimely

related https://github.com/toss/slash/pull/241

I have reviewed but I still think this change is not appropriate. Regardless of the intention of useQueryParam, someone may have already used it like:
const productIds = useQueryParam<string[]>('productIds');
const parsedProductIds = productIds.map((id) => parseInt(id));

I understand that you want to contribute, but It would be better to keep the previous implementation.

Even in this case, there is an error when used with parser.
If url is https://example.com?productIds=1&productIds=2
const productIds = useQueryParam<string[]>('productIds', {
  parser: (productIds) => { // productIds will be ["1", "2"], but typely this is string... then, what role is parser?
    /* we can't use productIds.map((id) => parseInt(id)) */
  }
});

You can see example typely invalid example or runtimely invalid example
typely invalid (https://example.com?productIds=1&productIds=2)

runtimely invalid (https://example.com?productIds=1&productIds=2)

Even though there are clearly edge cases in the code, It is difficult to understand why Pull Requests or other code suggestion cannot be accepted only because it is a BREAKING CHANGE. So what is BREAKING CHANGE for and why do other libraries do major bumps? If necessary, I think we should proceed with a BREAKING CHANGE.
Originally posted by @manudeli in https://github.com/toss/slash/issues/241#issuecomment-1585147140

If BREAKING CHANGE is difficult, creating an API with a new name, treating the API as deprecated, and gradually removing it seems like a way.

https://github.com/toss/slash/pull/241#issuecomment-1585189795


[Bug]: Plugin "@netlify/plugin-nextjs" internal error: ENOENT: no such file or directory with runtime 5.0.0-beta.7

Summary
I'm having an issue with getting our site to run on the beta runtime. We are currently on next 13.4.1 and testing with 5.0.0-beta.7
We are using turborepo for this site and have custom build settings in our netlify.toml to reflect that
Once committed, the deploy will build successfully and then fail after the onBuild with the following error:
11:15:26 AM: (@netlify/plugin-nextjs onEnd completed in 4ms)
11:15:27 AM:
11:15:27 AM: Plugin "@netlify/plugin-nextjs" internal error
11:15:27 AM: ────────────────────────────────────────────────────────────────
11:15:27 AM:
11:15:27 AM:   Error message
11:15:27 AM:   Error: ENOENT: no such file or directory, open "/opt/build/repo/apps/web/required-server-files.json"
11:15:27 AM:
11:15:27 AM:   Plugin details
11:15:27 AM:   Package:        @netlify/plugin-nextjs
11:15:27 AM:   Version:        5.0.0-beta.7
11:15:27 AM:   Repository:     git+https://github.com/netlify/next-runtime-minimal.git
11:15:27 AM:   npm link:       https://www.npmjs.com/package/@netlify/plugin-nextjs
11:15:27 AM:   Report issues:  https://github.com/netlify/next-runtime-minimal/issues
11:15:27 AM:
11:15:27 AM:   Error location
11:15:27 AM:   In "onBuild" event in "@netlify/plugin-nextjs" from netlify.toml and package.json
11:15:27 AM:
11:15:27 AM:   Error properties
11:15:27 AM:   {
11:15:27 AM:     errno: -2,
11:15:27 AM:     code: "ENOENT",
11:15:27 AM:     syscall: "open",
11:15:27 AM:     path: "/opt/build/repo/apps/web/required-server-files.json"
11:15:27 AM:   }
11:15:28 AM: Failed during stage "building site": Build script returned non-zero exit code: 4
11:15:28 AM: Build failed due to an internal system error: Build script returned non-zero exit code: 4
11:15:28 AM: Failing build: Failed to build site
11:15:28 AM: Finished processing build request in 4m29.05s

For what its worth, I did attempt to submit an issue on the requested links in that error, but was getting a 404.
I am wondering if the missing file is related to the monorepo?
Thanks!
A link to a reproduction repository
https://app.netlify.com/sites/workhuman-next/deploys/65c25a12e9e90f000874c61a
Expected Result
No internal errors + build to deploy successfully
Actual Result
Plugin "@netlify/plugin-nextjs" internal error
Steps to reproduce

Update to 5.0.0-beta.7 following documentation provided by Netlify Support
Build

Next Runtime version
5.0.0-beta.7
Is your issue related to the app directory?

[ ] Yes, I am using the app directory

More information about your build

[ ] I am building using the CLI
[X] I am building using file-based configuration (netlify.toml)

What OS are you using?
Mac OS
Your netlify.toml file

`netlify.toml`
# Settings in the [build] context are global and are applied to
# all contexts unless otherwise overridden by more specific contexts.
[build]
  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  base = "apps/web"

  # The ignore command provides custom, programmatic control over builds.
  # We use this to check if the respective site has been changed or not.
  ignore = "npx turbo-ignore"

  # Directory that contains the deploy-ready HTML files and
  # assets generated by the build. This is relative to the base
  # directory if one has been set.
  # publish = ".next"

  # Turborepo build command. We exit the base folder to make sure we can
  # access the root directory where our turbo config is.
  command = "cd ../.. && npx turbo run build --filter=web..."

[functions]
  included_files = [

    "!../../node_modules/sharp/vendor/*",

    "!../../node_modules/@esbuild/*",

    "!../../node_modules/@esbuild/*",

    "!../../node_modules/@swc/core-linux-x64-musl/*",

    "!../../node_modules/@swc/core-linux-x64-gnu/*",

    "!../../node_modules/@swc/core-darwin-x64/*",

    "!../../node_modules/sharp/vendor/8.14.5/linux-x64/lib/libvips-cpp.so.42",

    "!../../node_modules/@esbuild/linux-x64/bin/esbuild",

    "!../../node_modules/esbuild/lib/downloaded-@esbuild-linux-x64-esbuild",

    "!../../node_modules/esbuild/lib/downloaded-@esbuild-linux-x64-esbuild",

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
  ]

[[redirects]]
  force = true
  from = "/munchkin.js"
  status = 200
  to = "https://862-jiq-698.mktoweb.com/munchkin.js"

[[redirects]]
  force = true
  from = "/uc.js"
  status = 200
  to = "https://consent.cookiebot.com/uc.js"

[[redirects]]
  force = true
  from = "/heap.js"
  status = 200
  to = "https://cdn.heapanalytics.com/js/heap-XXXXXX.js"

[[redirects]]
  force = true
  from = "/uwt.js"
  status = 200
  to = "https://static.ads-twitter.com/uwt.js"

[[edge_functions]]
  function = "add-slash"
  path = "/*"

[[plugins]]
  package = "@netlify/plugin-nextjs"

Your public/_redirects file

`_redirects`
# Paste content of your `_redirects` file here

Your next.config.js file

`next.config.js`
/** @type {import('next').NextConfig} */
const nextI18nConfig = require('./next-i18next.config');

module.exports = {
  swcMinify: true,
  reactStrictMode: true,
  transpilePackages: [
    '@workhumanweb/ui',
    '@workhumanweb/api',
    '@workhumanweb/algolia-indexing-job',
    '@workhumanweb/common-i18n',
  ],
  eslint: {
    dirs: ['.'],
  },
  poweredByHeader: false,
  trailingSlash: true,
  // experimental: {
  //   adjustFontFallbacks: true,
  // },
  experimental: {
    forceSwcTransforms: true,
    // nextScriptWorkers: true,
    scrollRestoration: true,
  },
  api: {
    bodyParser: false,
  },
  images: {
    // dangerouslyAllowSVG: true,
    contentSecurityPolicy: "default-src 'self'; script-src 'none'; sandbox;",
    domains: [
      'www.workhuman.com',
      'www.workhumanpreprod.com',
      'images.ctfassets.net',
      'netlify.app',
    ],
    remotePatterns: [
      {
        protocol: 'https',
        hostname: 'images.ctfassets.net',
      },
    ],
  },
  env: {
    ...
  },
  i18n: nextI18nConfig.i18n,
  async headers() {
    ...
  },
  async rewrites() {
    return [
      {
        source: '/events/spotlight/:series',
        destination: '/spotlight-landing/:series',
      },
    ];
  },
};

Builds logs (or link to your logs)

Build logs
[Link to Logs](https://app.netlify.com/sites/workhuman-next/deploys/65c25a12e9e90f000874c61a)

Function logs

Function logs
# Paste logs here

.next JSON files

generated .next JSON files
# Paste file contents here. Please check there isn't any private info in them
# You can either build locally, or download the deploy from Netlify by clicking the arrow next to the deploy time.

11:15:27 AM:   Error: ENOENT: no such file or directory, open "/opt/build/repo/apps/web/required-server-files.json"

I believe the file does exist at /opt/build/repo/apps/web/.next/required-server-files.json
I don't have a solution yet, but maybe someone working on the plugin knows where this bug is.  Will dig a bit

I think this might have to do with netlify.toml values, but I haven't figure out the right ones yet

Based on testing in the codebase that @caomicc originally posted it appears that there's a bug in the 5.x-beta.7 and 8 version of the @netlify/plugin-nextjs that it assumes the wrong path.  From a team member

It assumes that standalone files should live in .next/standalone/.next/ but the correct path is .next/standalone/apps/web/.next/ in monorepo configuration. I've tired many different configuration options without any success.

We are now experiencing the same issue with it not being able to find the required-server-files.json. We tried pinning the version to "5.0.0-beta.4" but that did not help. We have also tried version "5.0.0-rc.0" with no luck.

Hey @jesseday and @SilverFox70.
We just released an early access version that should fix many issues.
Can you try installing @netlify/plugin-nextjs@rc and let us know if you still have the problems?
Thanks!

@MarcL Hi, I too am still having this issue.
Once trying this update to rc, which installs ^5.0.0-rc.4, I can no longer install any packages and am getting a new error:
9:41:45 AM: npm ERR! code ERR_INVALID_ARG_TYPE
9:41:45 AM: npm ERR! The "from" argument must be of type string. Received undefined
9:41:45 AM: npm ERR! A complete log of this run can be found in: /opt/buildhome/.npm/_logs/2024-03-13T13_41_08_575Z-debug-0.log
9:41:45 AM: Error during npm install
9:41:45 AM: Failing build: Failed to install dependencies
9:41:45 AM: Failed during stage "Install dependencies": dependency_installation script returned non-zero exit code: 1

Reverting to ^5.0.0-beta.9 afterwards does continue to install dependencies however fails on the same spot in the build.

@caomicc @jesseday @SilverFox70 - We've released v5 now and started rolling it out. Can you remove your pinned dependencies and build and redeploy and check it now?

I ran into something similar, and I think I have a workaround.
TL;DR: instead of reading from .next/, read from .netlify/.next/. This feels like a temporary workaround and probably not the ideal solution.
  const path = require('path');

  module.exports = {
    async onPostBuild({ constants, inputs }) {
      const { PUBLISH_DIR } = constants; // where PUBLISH_DIR === '.next'
      const mediaDir = 'static/media';
<     const cwd = path.join(PUBLISH_DIR, mediaDir);
>     const cwd = path.join('.netlify', PUBLISH_DIR, mediaDir);
      // ...
    }
  };

After upgrading to v5 (currently on 5.1.2), I noticed our static assets weren't being uploaded to our CDN. We use a custom onPostBuild hook to grab all the nextJS assets and push them to the CDN.
Some digging, I found that .next directory now looks more like the built public directory. More digging and I found that .netlify/.next looks like the original .next output.
.next/
  _next/
  assets/
  audio/
  cms/
  downloads/
  img/
  android-chrome-192x192.png
  android-chrome-256x256.png
  apple-touch-icon.png
  browserconfig.xml
  favicon-16x16.png
  favicon-32x32.png
  favicon-512x512.png
  favicon.ico
  robots.txt
  site.webmanifest
  sitemap-0.xml
  sitemap.xml

.netlify/.next/
  cache/
  server/
  standalone/
  static/
  BUILD_ID
  build-manifest.json
  export-marker.json
  images-manifest.json
  next-minimal-server.js.nft.json
  next-server.js.nft.json
  package.json
  prerender-manifest.js
  prerender-manifest.json
  react-loadable-manifest.json
  required-server-files.json
  routes-manifest.json
  trace

FWIW, when I inspect the root .next directory outside of the plugin, it looks correct, but when debugging inside of the plugin it is as noted above.

@dragons-library yes, I'd recommend just ensuring the Netlify plugin runs last, rather than relying on the internals of our build process.
Everyone else: in 5.1.2 there are some improvements to finding the build output on sites with custom setups, so if you have issues that relate to that then I'd recommend trying again to see if it now works


the funtion formatting is deprecated

I think we should use the
vim.lsp.buf.format { async = true }

This is re-introducing the same regression fixed in 134a291fecc5c1fa5d01e398d2d6ef03b3dc5149

Think it requires Nvim 0.8. So yeah should revert this until format() is in stable release.


[RFC] Add integer division operator

I have a lot of stuff where I need integer division and currently have to write stuff like this:
var rem = a % b
var div = (a/b).floor

The second line is relatively tedious and, looking back through old issues to see whether this had been proposed before, I see that @munificent admitted as much in #127.
So, would it be worthwhile to have an integer division operator as well as an ordinary division operator?
If we did, then there would be a number of questions to answer:

What symbol should be used for the operator?
How exactly should it work?
What should its precedence be?
Should it be overloadable?

Taking each of these in turn:

The only integer division operators I can recall seeing in other languages are \ in BASIC
and // in Python. I have no particular preference between them though I suspect the former would be easier to parse so I'll use that in the remainder of this post.

Given that Wren doesn't have integers as such, I think we'd want the operator to apply to any two Nums whether they're integers of not. There's then the question of negative numbers. Which of the following would be best:

var a = -4.6
var b = 2
var c = a \ b  // -2 i.e. (a/b).truncate
var d = a \ b  // -3 i.e. (a/b).floor

I think I'm right in saying that BASIC typically uses truncated and Python floored division.

I can't see any argument for it being other than the same as / i.e. precedence level 3 with (obviously) left associativity.

Sure, why not.

A slightly awkward point is that the % operator returns a result which has the same sign as the numerator. So, here we have:
var a = -4.6
var b = -2
var c = a % b
System.print(c) // -0.6, not +0.6, because a is negative

Although it would be nice for the integer division and modulo operators to be consistent with each other, I think it would seem anti-intuitive to most people if a \ b were -2 in this example.
As a further alternative would it be best to just add an integer division method to the Num class (perhaps a.div(b) or a.idiv(b)) rather than an operator which would be easier to implement?
As always, comments would be appreciated.

It is not harder to parse // than \, but we can't use it since Wren uses // for line comments (Python uses # for that so it could).

It is not harder to parse // than \, but we can't use it since Wren uses // for line comments (Python uses # for that so it could).

Good point, I'd forgotten about that.

Good point, I'd forgotten about that.

I would not use \ for various reasons, main one being that it is usually
used to break a line in script languages, and we will need that to expand
the repl at some point.

I would not use \ for various reasons, main one being that it is usually
used to break a line in script languages, and we will need that to expand
the repl at some point.

Do you mean  a.floor(b)  would be equivalent to (a/b).floor ?
If so, I'm not too keen on it as I think it would be potentially confusing.
I'd prefer a.div(b) or we could even have a div b as there's a precedent for using a word as an operator (namely is). However, as div would need to be a reserved word, the latter would not be backwards compatible so I think we can rule it out immediately.

Do you mean  a.floor(b)  would be equivalent to (a/b).floor ?
If so, I'm not too keen on it as I think it would be potentially confusing.
I'd prefer a.div(b) or we could even have a div b as there's a precedent for using a word as an operator (namely is). However, as div would need to be a reserved word, the latter would not be backwards compatible so I think we can rule it out immediately.

Comments so far on this issue have been very useful and I've come to the conclusion that an integer division operator is not going to be practicable because there simply isn't a symbol available which is suggestive of division.
To my mind that just leaves us with adding an a.div(b) or a.idiv(b) method to the Num class. Personally, I prefer the former as simple sugar for (a/b).truncate i.e. always round towards zero.
So I'll leave the issue open for any further thoughts on the matter.

Comments so far on this issue have been very useful and I've come to the conclusion that an integer division operator is not going to be practicable because there simply isn't a symbol available which is suggestive of division.
To my mind that just leaves us with adding an a.div(b) or a.idiv(b) method to the Num class. Personally, I prefer the former as simple sugar for (a/b).truncate i.e. always round towards zero.
So I'll leave the issue open for any further thoughts on the matter.

If you go that route than .quo should be a synonym of .truncate if I'm
not mistaken. This is why I proposed the version of .truncate(_) an
.floor(_) to not add extra names in the first place.

If you go that route than .quo should be a synonym of .truncate if I'm
not mistaken. This is why I proposed the version of .truncate(_) an
.floor(_) to not add extra names in the first place.

Yes, a.quo(b) would be synonymous with (a/b).truncate.
However, the reason I was suggesting this in the first place was so that it would be shorter and, hopefully, clearer to write.
I don't think it's clear what  a.truncate(b) would mean as there's no implication of division and it isn't short either.  Personally, I'd prefer to just live with what we have rather than to introduce that.

Yes, a.quo(b) would be synonymous with (a/b).truncate.
However, the reason I was suggesting this in the first place was so that it would be shorter and, hopefully, clearer to write.
I don't think it's clear what  a.truncate(b) would mean as there's no implication of division and it isn't short either.  Personally, I'd prefer to just live with what we have rather than to introduce that.

OK, I think I'm going to close this issue as a new operator clearly isn't viable and a new method (or overloading an existing one) doesn't seem worth the effort. At least what we already have, even if verbose, is clear and you can round it how you like.
It's something that's been bugging me for a while so I feel better for having discussed it. Hopefully,  commenters will agree that it was at least worth giving it an airing :)

OK, I think I'm going to close this issue as a new operator clearly isn't viable and a new method (or overloading an existing one) doesn't seem worth the effort. At least what we already have, even if verbose, is clear and you can round it how you like.
It's something that's been bugging me for a while so I feel better for having discussed it. Hopefully,  commenters will agree that it was at least worth giving it an airing :)


[BUG] some system indices growing indefinitely

Describe the bug
I have observed that for some system indices such as below, may grow indefinitely:

.opendistro-ism-config
.opendistro-job-scheduler-lock

Even if they continue to grow, I see, they have a lot of deleted documents.
Related component
Indexing
To Reproduce
Observation using the metric: elasticsearch_indices_store_size_bytes_total
Expected behavior
The system indices should not grow indefinitely
Additional Details
No response

@dblock Why was this transferred to indexing?
System indices belong to different plugins, first belongs to ISM and second belongs to job-scheduler. Also, I don't see enough details on how it was concluded that index growth was not organic but was potentially a bug.

@soorajambadi You may need to look into what data these indices contain and whether growth in data makes sense  or it is a leak somewhere. For e.g., in a cluster where there are too many indices getting rolled over and created, growth in data would make sense.

@vikasvb90 I did some analysis to check if the growth is related to increase in the number of indices in the cluster. To my limited knowledge the system index .opendistro-job-scheduler-lock stores (in my case) lock information of different jobs. This has following fields: job_index_name, job_id, lock_time, lock_duration_seconds, and released. When I did:
GET /.opendistro-job-scheduler-lock/_search
I got 39 hits and all has job_index_name value .opendistro-ism-config, but different job_id values.
When I did GET /.opendistro-ism-config/_search  I got the same number of managed indices with "_id" corresponds to previously found job_id values.
In summery, the indices are not growing and the .opendistro-job-scheduler-lock index contains number of documents corresponding to the number of non-system indices.
Given the index number is constant during the time of measure, the .opendistro-job-scheduler-lock size still grows, which brings the suspicion. Attaching the 24 hour size growth with constant index number (39):

Looks like some issue with the job scheduler plugin, as it's system index is the growing.
@opensearch-project/admin  Can we move it to https://github.com/opensearch-project/job-scheduler?

@soorajambadi @dhwanilpatel What version of OpenSearch are you running?

Even if they continue to grow, I see, they have a lot of deleted documents.

Found a forum post that may be relevant: https://forum.opensearch.org/t/indices-become-bloated-not-deleting-deleted-docs-and-old-doc-versions/11019
Is it possible that the index.soft_deletes.retention_lease.period setting is set to a non-default value?

@cwperks thanks for the details. We are running OpenSearch 1.3.x, although there are plans for an upgrade.
Indeed I can see a lot of documents marked for deletion and it grows. I see after few days it drops to a lower value (Zig-zag nature) which purges deleted docs during a merge.
Since the root cause is clear and is not a bug but how OpenSearch works, I think the issue can be closed.