Cannot connect to cloudcutterflash AP on Raspberry Pi

I've made several attempts on this device. It got an "A-" AP, so I guess that means it has been "cut"? This is a TreatLife SL-10 RGB bulb, which Tuya app showed as version 1.3.21 (checked before I started). Previous reports on these bulbs showed different versions (e.g. 1.0.8), which I tried first without the exploit succeeding and it seemed to work on a similar "Globe" bulb with that version number.
At the cloudcutterflash portion of the process, the bulb doesn't complete the connection to the cloudcutterflash AP. I tried with my phone also and it says abcdabcd is the wrong password.
Verbose terminal output is here.
I attempted to connect to the bulb using the Tuya app ("add" function), but that didn't find the bulb, either. I'm assuming it should still recognize the device's Bluetooth pairing?
I successfully converted/programmed several tasmota devices on this Raspberry Pi recently (tuya-convert).

abcdabcd is the correct password ( see https://github.com/tuya-cloudcutter/tuya-cloudcutter/blob/4aa91be237b811e24c71ab15eb8bb6b009d093b6/src/setup_apmode.sh#L42 for where we set it ).  Make sure you are in range (AP mode is very weak) and your phone keyboard is not changing case or trying to autocorrect.
Tuya app should see the device in bluetooth pairing mode, but will not actually add it (but it should attempt to make it connect to cloudcutterflash), but I still highly recommend using the slow-blink method instead of the bluetooth method.
Are you already following the directions at https://github.com/tuya-cloudcutter/tuya-cloudcutter/wiki/FAQ#my-device-gets-stuck-after-dhcp-what-can-i-do ?

Thanks for the quick response!
My RaspberryPi and the bulb are both on my desk. Yes, I've read through the troubleshooting page and attempted the section you mentioned.
I got a little farther today. The bulb got to DHCPACK stage and IP address assigned, but the firmware still wasn't getting downloaded. I thought the difference might be because I was using a wifi dongle (wlan1) on my pi but then I also got to DHCPACK using onboard wifi (wlan0) like before, so maybe not.
My phone was able to connect to the AP when it was running on wifi dongle, but not onboard wifi. Both of the Tuya apps I have on my phone told me the cloudcutterflash network was not usable for adding a device when using either quick flash or slow flash (I presume because of lack of internet?), and the app would not let me continue past that point.
Another thing I tried was using a second Raspberry Pi to connect to the cloudcutter pi instead of my phone. It was constantly connected/disconnected when cloudcutter pi was using wlan0, but did get a stable connection when it was using wlan1. The Tuya app runs on Android unfortunately, so hmmm.
I also tried an older phone of mine, but it wouldn't connect to the AP and is too old for the apps, anyway.
Here is a bit of progress on the connection...
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> DHCPACK(wlan0) <IP_ADDRESS> a8:80:55:e9:06:89 wlan0
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> requested options: 1:netmask, 3:router, 28:broadcast, 6:dns-server
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> next server: <IP_ADDRESS>
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  1 option: 53 message-type  5
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option: 54 server-identifier  <IP_ADDRESS>
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option: 51 lease-time  12h
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option: 58 T1  6h
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option: 59 T2  10h30m
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option:  1 netmask  <IP_ADDRESS>
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option: 28 broadcast  <IP_ADDRESS>
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option:  3 router  <IP_ADDRESS>
Oct 26 03:08:20 dnsmasq-dhcp[15]:<PHONE_NUMBER> sent size:  4 option:  6 dns-server  <IP_ADDRESS>

All kinds of output similar to this from the second Raspberry Pi:
Oct 26 03:52:10 dnsmasq[15]: query[A] www.googleapis.com from <IP_ADDRESS>
Oct 26 03:52:10 dnsmasq[15]: config www.googleapis.com is <IP_ADDRESS>
SSL Error on 16 ('<IP_ADDRESS>', 33992): [SSL: NO_SHARED_CIPHER] no shared cipher (_ssl.c:1129)
SSL Error on 16 ('<IP_ADDRESS>', 33994): [SSL: NO_SHARED_CIPHER] no shared cipher (_ssl.c:1129)

Not much I can advise, these types of issues are outside the control of CloudCutter.
The only two suggestions I have are

Try alternate AP channels, by default, it is on channel 1 in case it is an interference issue.  This can be adjusted in src/setup_apmode.sh
Specifically try the 2022-04-04 Bullseye build of RaspbianOS.

Question I had at this point was, is there a way to send the firmware outside of the main script?

Nope, it specifically needs to go through a progression of actions, it's not a single step to firmware upgrade.

Then I downloaded the old 2022-04-04 Bullseye you suggested and set it up fresh per HOST_SPECIFIC_INSTRUCTIONS.md and had success on the first try.

Excellent!  I've remained on that version mostly just out of lazyness, but I might have to start a new install to see if I can actually finally reproduce the stall issue, in which case I might finally be able to directly test it rather than through proxy.  The thing that baffles me though is this is in a container, so I wonder if it's an issue with docker somehow, as the versions of things running within should be consistent.

Now I am trying to figure out how to get the device into Home Assistant / ESPHome, which seems to be insisting I need to connect the device using a cable.

No need for a cable.  If you have kickstart installed on it, you can compile your firmware and just upload it to the kickstart web interface.  After that if your firmware was compiled correctly, it will be discovered by HA automatically.
Can you please verify which version of Raspbian you had that was causing the problem?
Since switch RaspbianOS version resolved your issue, I'm closing this ticket, as that is the ultimate solution for the time being.

It was a pretty fresh instance, BUT it had tuya-convert and a "reverse" wifi password cracker I wrote to figure out what password I incorrectly entered into a tuya device (turns out I had entered it correctly after all, and had just forgotten that I enabled MAC filtering on my router). The latter used hostapd so maybe I left something badly configured, although the changes were really just to write different passwords into it and see if the device connected.
(32-bit)
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

$ uname -a
Linux 6.1.21-v7+ #1642 SMP Mon Apr  3 17:20:52 BST 2023 armv7l GNU/Linu                                               x

Many thanks to you @Cossid , I was able to flash both of my RGB lights and got them working with Home Assistant.
The final hurdles were leaped by following the steps in this video.
I'm not sure that the BK7231N variant firmware is the right one for these. The FCC number ends with WB8P, which according to this, indicates they have the WB8P chip in them, which according to various sources (#184, here, here), means the BK7231T variant firmware should be used.
I used the kickstart-bk7231n firmware with pinscan and confirmed that the yaml I got from ltchiptool had the right pin setup (The profile for the Globe 35798 A19 RGBCT v13.21 was not).
I don't know all the ins and outs, but it looks like the pins on T and N are the same but memory layout may be different, according to unofficial BK7231 datasheet, pinout, programming, specification, wiki (BK7231T, BK7231N). Beken doesn't seem to have datasheets for these chips on their website.

It is not uncommon for the FCC information to be outdated, as these cheap manufacturers change the modules unfortunately too frequently without updating documentation.
Follow what CloudCutter and Kickstart are instructing, as it is absolutely always 100% correct.  It tracks the chip based on the exploit used, and you can't exploit a BK7231T device with a BK7231N profile or vice versa.  If you still have Kickstart installed or put it back on, you can use ltchiptool to export the storage data under the upk2esphome tab (and post it here, we can possibly add direct device support) and if the device has a upk section, it will tell you what chip is actually in the device in the device under the "module" property.
Luckily, CloudCutter and ESPHome will not let you flash firmware for the wrong chip unless you try really really hard.

If I get some more of these lights, I will post the data out of ltchiptool. The family seemed to be enjoying the one I had working yesterday, so I may get some more.

You actually don't need to reinstall kickstart, you can add the following:
external_components:
  source: github://libretiny-eu/esphome-kickstart
  components:
    - hub_api
  refresh: 10 min

hub_api:

to your config, and you can use ltchiptool just the same as if you had kickstart installed.  (then you can remove that block and re-upload when done)

Is this the right data? This shows CB8P.
ltchiptool shows this warning, but only 4 pins actually do anything per my testing with pinscan one at a time. Also, ESPHome has a page for rgbw, rgbww, rgbct but not rgbcw, so I don't know what that means, or if it can be matched to something better than rgbw.
W: UPK: Module 'cmod': rgbcw doesn't match platform rgbw
{
    "gw_bi": {
        "uuid": "p5a384pwXsUq",
        "psk_key": null,
        "auth_key": "mdHFG6j2givJ92za",
        "ap_ssid": "A",
        "ap_passwd": null,
        "country_code": null,
        "bt_mac": null,
        "bt_hid": null,
        "prod_test": false,
        "fac_pin": null
    },
    "user_param_key": {
        "Jsonver": "1.0.0",
        "b_lv": 1,
        "b_pin": 9,
        "brightmax": 100,
        "brightmin": 5,
        "c_lv": 1,
        "c_pin": 24,
        "cagt": 5,
        "category": "0505",
        "cmod": "rgbcw",
        "colormax": 100,
        "colormin": 10,
        "crc": 51,
        "cwtype": 0,
        "defbright": 100,
        "defcolor": "c",
        "deftemp": 0,
        "dmod": 0,
        "g_lv": 1,
        "g_pin": 8,
        "gmkb": 60,
        "gmkg": 60,
        "gmkr": 80,
        "gmwb": 75,
        "gmwg": 70,
        "gmwr": 100,
        "mixway": 3,
        "module": "CB8P",
        "notdisturb": 0,
        "onoffmode": 1,
        "pmemory": 1,
        "prodagain": 1,
        "pwmhz": 2000,
        "r_lv": 1,
        "r_pin": 7,
        "remdmode": 0,
        "rgbt": 5,
        "rstbr": 50,
        "rstcor": "c",
        "rstnum": 3,
        "rsttemp": 0,
        "tempmix": [
            [
                "T18",
                "000",
                "000",
                "000",
                "3e8"
            ],
            [
                "T22",
                "000",
                "0ca",
                "029",
                377
            ],
            [
                "T27",
                "000",
                151,
                "045",
                "32a"
            ],
            [
                "T30",
                "000",
                "1b1",
                "05c",
                "2f1"
            ],
            [
                "T35",
                "000",
                236,
                "07e",
                "2a7"
            ],
            [
                "T40",
                "000",
                "2b3",
                "0a3",
                268
            ],
            [
                "T45",
                "000",
                310,
                "0c2",
                231
            ],
            [
                "T50",
                "000",
                352,
                "0dc",
                201
            ],
            [
                "T57",
                "000",
                385,
                "0f2",
                "1d6"
            ],
            [
                "T60",
                "000",
                "39d",
                100,
                "1b0"
            ],
            [
                "T65",
                "000",
                "39e",
                106,
                "18c"
            ],
            [
                "T70",
                "000",
                "3a4",
                "10d",
                "16e"
            ],
            [
                "T75",
                "000",
                "3b2",
                115,
                "15f"
            ],
            [
                "T80",
                "000",
                "3c3",
                "11e",
                154
            ],
            [
                "T90",
                "000",
                "3d6",
                127,
                "14a"
            ],
            [
                "T100",
                "000",
                "3e8",
                130,
                "13f"
            ]
        ],
        "title20": 1,
        "wfcfg": "spcl",
        "wfct": 3
    },
    "gw_di": {
        "abi": 0,
        "id": "QLjZPBjM2ISeN8lr3O1i",
        "swv": "1.3.21",
        "bv": "40.00",
        "pv": "2.2",
        "lpv": "3.4",
        "pk": "key8u54q9dtru5jw",
        "firmk": "key8u54q9dtru5jw",
        "cadv": "1.0.5",
        "cdv": "1.0.0",
        "dev_swv": "1.3.21",
        "s_id": "000004igo3",
        "dtp": 0,
        "sync": 0,
        "attr_num": 1,
        "mst_tp_0": 9,
        "mst_ver_0": "1.3.21",
        "mst_tp_1": 0,
        "mst_ver_1": null,
        "mst_tp_2": 0,
        "mst_ver_2": null,
        "mst_tp_3": 0,
        "mst_ver_3": null
    },
    "tls_ca_cnt": 0,
    "gw_wsm": {
        "nc_tp": 4,
        "ssid": "Y2xvdWRjdXR0ZXJmbGFzaA==",
        "passwd": "YWJjZGFiY2Q=",
        "md": 0,
        "random": 0,
        "wfb64": 1,
        "stat": 2,
        "token": "AAAAAA",
        "region": "AA",
        "reg_key": null,
        "dns_prio": 0
    },
    "wf_start_md": 3,
    "is_stride": 0,
    "gw_ai": {
        "key": "GU1RUzD7tRJMAzVO",
        "lckey": "xkkdag3WvNgH6HOu",
        "h_url": "http://<IP_ADDRESS>/d.json",
        "h_ip": "<IP_ADDRESS>",
        "hs_url": null,
        "hs_ip": null,
        "hs_psk": "https://a3.tuyaus.com/d.json",
        "hs_psk_ip": "<IP_ADDRESS>",
        "mqs_url": null,
        "mqs_ip": null,
        "mq_url": "<IP_ADDRESS>:1883",
        "mq_ip": "<IP_ADDRESS>",
        "ai_sp": null,
        "ai_sp_ip": null,
        "mq_psk": "m2.tuyaus.com:8886",
        "mq_psk_ip": "<IP_ADDRESS>",
        "lp_url": "baal.tuyaus.com",
        "lp_ip": "<IP_ADDRESS>",
        "time_z": "+01:00",
        "s_time_z": "[[1648342800,1667091600],[1679792400,1698541200]]",
        "wx_app_id": null,
        "wx_uuid": null,
        "dy_tls_m": 1,
        "cloud_cap": 1025,
        "psk21_key": null
    },
    "000004igo3": [
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "type": "bool"
            },
            "id": 20
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "range": [
                    "white",
                    "colour",
                    "scene",
                    "music"
                ],
                "type": "enum"
            },
            "id": 21
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "min": 10,
                "max": 1000,
                "scale": 0,
                "step": 1,
                "type": "value"
            },
            "id": 22
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "min": 0,
                "max": 1000,
                "scale": 0,
                "step": 1,
                "type": "value"
            },
            "id": 23
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 24
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 25
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "min": 0,
                "max": 86400,
                "scale": 0,
                "step": 1,
                "type": "value"
            },
            "id": 26
        },
        {
            "type": "obj",
            "mode": "wr",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 27
        },
        {
            "type": "obj",
            "mode": "wr",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 28
        },
        {
            "mode": "rw",
            "id": 30,
            "type": "raw"
        },
        {
            "mode": "rw",
            "id": 31,
            "type": "raw"
        },
        {
            "mode": "rw",
            "id": 32,
            "type": "raw"
        },
        {
            "mode": "rw",
            "id": 33,
            "type": "raw"
        },
        {
            "mode": "rw",
            "id": 209,
            "type": "raw"
        },
        {
            "mode": "rw",
            "id": 210,
            "type": "raw"
        }
    ],
    "000004f4gf": [
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "type": "bool"
            },
            "id": 20
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "range": [
                    "white",
                    "colour",
                    "scene",
                    "music"
                ],
                "type": "enum"
            },
            "id": 21
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "min": 10,
                "max": 1000,
                "scale": 0,
                "step": 1,
                "type": "value"
            },
            "id": 22
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "min": 0,
                "max": 1000,
                "scale": 0,
                "step": 1,
                "type": "value"
            },
            "id": 23
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 24
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 25
        },
        {
            "type": "obj",
            "mode": "rw",
            "property": {
                "min": 0,
                "max": 86400,
                "scale": 0,
                "step": 1,
                "type": "value"
            },
            "id": 26
        },
        {
            "type": "obj",
            "mode": "wr",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 27
        },
        {
            "type": "obj",
            "mode": "wr",
            "property": {
                "type": "string",
                "maxlen": 255
            },
            "id": 28
        },
        {
            "mode": "rw",
            "id": 31,
            "type": "raw"
        },
        {
            "mode": "rw",
            "id": 32,
            "type": "raw"
        }
    ],
    "em_sys_env": "bk7231n",
    "mf_test_close": true
}

Yep, that is correct, and confirms they swapped the WB8P with the pin compatible newer CB8P, which is indeed BK7231N
I'll add a new cloudcutter profile shortly.

ltchiptool shows this warning, but only 4 pins actually do anything per my testing with pinscan one at a time. Also, ESPHome has a page for rgbw, rgbww, rgbct but not rgbcw, so I don't know what that means, or if it can be matched to something better than rgbw.
W: UPK: Module 'cmod': rgbcw doesn't match platform rgbw

Yeah, this is a non-standard bulb.  It is actually a rgbw bulb, and one of your white channels is emulated by adding in color LEDs in white mode.  Unfortunately, this can be a little tricky to do with ESPHome.  That is the the "tempmix" property was doing, it was replacing a white channel with defined color channels and a percentage of the white channel.


Save changes when switching between versions

I just lost all of my changes after working on v2 and realizing that I need to switch to 3. Maybe I should have seen it coming, but I don't expect a drop down list to redirect me to another page.

Sorry, no idea what you're talking about? This is only version 2.


Avalanche Support for Human Protocol

Description
PR that aims to add support of Avalanche mainnet and testnet, as discussed in #137
Summary of changes
Added network details to escrow-dashboard and CONTRACT_LIST.md

Where is fortune running?

@posix4e I had deployed the fortune app locally, that ran on

Fujinet (Avalanche testnet)
Avalanche C-Chain

Testnet Video Demo
Mainnet Video Demo

@posix4e everything is on this message on this avalanche issue : https://github.com/humanprotocol/human-protocol/issues/137#issuecomment-1372085275

@hrik2001 please verify the contracts in both testnet and mainnet scanners

Fortune works for both testnet and mainnet. Waiting for changes requested

Hi, thanks @foufrix @portuu3, had to run some errands hence couldn't do the changes. Changes will be done in an hour , thanks a lot :)

@hrik2001 contracts still not verified

@foufrix I cannot be approved and merged yet, if we merge now we will break the dashboard since the subgraphs are not done.
@hrik2001 please can you implement subgraphs too?

@portuu3 sure, so essentially I have to add configs for avalanche at human-protocol/packages/sdk/typescript/subgraph/config/ right?
Also you had said

contracts still not verified

I am not sure what you mean by this, how should I go about that?

Also you had said

contracts still not verified

I am not sure what you mean by this, how should I go about that?

If you click in the contract tab of the scanner you will have an option to verify it. Just click on it and go trough the process, it will ask you to upload the source code of the contracts you used.

@portuu3 sure, so essentially I have to add configs for avalanche at human-protocol/packages/sdk/typescript/subgraph/config/ right?

The section Add a new network in human-protocol/packages/sdk/typescript/subgraph/README.md should help you

@portuu3 just wanted to confirm, if I host the subgraph, there would be my username in the subgraph link and not human protocol, you guys are okay with that right?

@portuu3 just wanted to confirm, if I host the subgraph, there would be my username in the subgraph link and not human protocol, you guys are okay with that right?

@hrik2001 just do the changes in the repo and I can do the step for creating the subgraph

@portuu3 feel free to review


fix: correct dependency version

:tada: This PR is included in version 1.3.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Loading pre-trained embeddings

Hi!
Thank you for making this toolkit available!
I have a collection of pre-trained embeddings with different vector sizes, and I have a question regarding the word_emb_dim setting in the configuration file.
Let's say we load a 300-dimensional pre-trained embedding, and keep the default word_emb_dim value, which is 50. What will happen then? Will word_emb_dim be overridden?

Yes, it will be overwritten as the real pretrain embeddings size.

Thank you! :+1:


App generator: Ask for public folder and possibly colocated tests

As part of the app generator it would nice/complete if users were asked where to put the public/ folder. For example, I could put my source in web/src and my static assets in web/public.
Also, it would be nice (wishful thinking) if I were asked where tests should be located. The default could be test, but another option would be (colocated). Colocated would mean that tests would be located next to the file/module they were testing.

Thank you for the suggestion. Our goal with the generator is to pick sane defaults and keep the number of prompts as low as possible, especially when it is not relevant to other generators. Every new prompt will have to be implemented, tested, documented and released and also requires all tutorials  that are showing and/or explaining the prompts (including screenshots) to be updated.
Both changes you are suggesting can be made by editing a JSON file. The public folder can be changed by renaming it and updating the configuration value in config/default.json to the new path. Support for customizing the test folder has been added in #218. Tests will be generated into the folder specified inlib.test in the package.json.

Good points, but now you guys are pigeonholed into what you got. Is there anyway to automate some of the process in updating tutorials/guides/screenshots? I suppose there would only actually be a few steps that have the potential for automation. Anyway, thanks for the feedback @daffl.

Definitely possible but also probably a lot of work (I know of an open source project where it took a developer a month full time to implement something similar).
What I'm trying to say is that we'd like to avoid adding prompts for style choices and anything that only requires changing one or two lines of code (especially since outside of JSON files, modifying existing code programmatically is quite complicated and brittle).
I'm definitely going to write up a guide on how to fork and customize the generator for your own needs though and pull requests against the generator are always going to be considered of course.

Thanks for the insight. I figured forking would be the best thing for us. We'd be able to make modifications and be able to pull in changes from upstream and send pull requests for any issues we resolve.

Sounds good. Pull requests are always welcome 😄

@daffl Any progress in the guide on how to fork and customize the generator?


Update boto3 dependency to py311-boto3 for OPNsense 24

It looks like the py39 package collection is no longer present in 24.1, superseded by the py311 collection. Would it be possible to update the boto3 dependency to py311-boto3? I would be happy to make a PR for this, but I don't see the dependency listed anywhere in this repo, only appearing in the packagesite.yaml on repo.threatpatrols.com, which leads me to believe it's something only you can update.
Appreciate all the work you've put into this!

Hi @studioph have taken a look into this for you
Yes, this one got away from us as we did not notice the mid-cycle Python upgrade in 24.1 from Py39 to Py311 - totally our fault.
We've got a release builder with the latest 24.1 that includes py311 just now and the configsync update should be ready in a few hours.
To your note that there is no mention of boto3 in the package (ie in the Makefile) this is because we inject this value in the build process to accommodate python prefix changes such as in this case from py39- to py311-
Here's a lovely view of our internal build instances as we play catchup

os-configsync-0.4.35.pkg released to 24.1 stable branch, that resolves this issue.
A regular OPNsense update should make the 0.4.35 version available to install and/or upgrade.
Thanks for reporting.

Thank you for the update Nicholas!


Horse dismount bug

I am currently running

SpongeVanilla version:  1.12.2-7.1.6

Java version: 1.8.0_181

Operating System: Windows 10 Pro x64

Plugins/Mods: None

I have some bug with horse. Sometimes if dismount from horse, it disappears (i hear horse sounds) or teleports to start location. Also if I'm riding a horse, I can start teleport to start and back about 10 times. I think this is vanilla bug. I found a similar problem here: https://bugs.mojang.com/browse/MC-101247 and it fixed in 1.15 snapshot

SpongeVanilla version: 1.12.2-7.1.6

That build is 5 months old. Please try to reproduce with the latest version (there have been a few issues about disappearing entities not long ago).


Updates darc dependencies from the .NET Core Tooling Dev channel

Noticed that with the merge of PR #8864 (updating dependencies from arcade) the APIScan started failing on Razor Tooling Compliance pipeline.
Updating some of the .NET Core Tooling Dev in this PR meanwhile trying to understand why this is happening
@jjonescz @jaredpar

Is the Roslyn bump expected to fix the API Scan issue? Or is it unrelated?

Thanks for the heads up, I assumed the PR updating the packages would be harmless. Closing since it might fail integration tests.
Was mainly looking to get feedback this way and get clarification on next steps.

I'll reopen by just updating the arcade SDK to newer version, leaving out the update to roslyn packages


[BEAM-11275] Support remote package download from remote filesystems in Stager

Support download of remote packages with URI schemes supported in apache_beam.io.filesystems in stager.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @ibzib
Could you help me review this at your convenience? Thanks
Do we need integration tests to test that pulling from GCS indeed works/ the URL typed ArtifactInformation didn't break anything else?

@calvinleungyk - Would you be able to respond to open comments?

Hey @aaltay, sorry was on vacation and don't have much internet access. Will respond to the comments in detail and make the changes after tomorrow morning.

retest this please

@ibzib I'm also trying to run integration tests according to
clear && clear && ./scripts/run_integration_test.sh --test_opts "--tests=apache_beam.runners.dataflow.dataflow_exercise_streaming_metrics_pipeline_test --nocapture" --project <gcs_project_name> --gcs_location gs://<bucket_path> --kms_key_name "" --streaming true --worker_jar <Path_To_Jar_Binary>
Does GCP project apache-beam-testing have an appropriate bucket and is there a worker_jar to run this?
Primarily, the stager now doesn't download and stage the remote packages and only write the package name to the extra_packages.txt file. I'm not certain where to test the behavior where the worker reads the txt file and decides what to install.

I'd recommend running integration tests using Gradle, since it fills in a lot of that boilerplate.
./gradlew :sdks:python:test-suites:dataflow:py38:postCommitIT
Then you can see which arguments it uses and change the ones you need.

Hi @ibzib, I've got a question about common_urns.artifact_types.URL.urn. From the code path sdks/python/container/boot.go -> sdks/go/pkg/beam/artifact/materialize.go, it doesn't seem like URL artifacts are being retrieved at worker startup. Do I need to add to Golang files to make this work?

Hi @ibzib, I've got a question about common_urns.artifact_types.URL.urn. From the code path sdks/python/container/boot.go -> sdks/go/pkg/beam/artifact/materialize.go, it doesn't seem like URL artifacts are being retrieved at worker startup. Do I need to add to Golang files to make this work?

Can you share more details about the problem you're facing?
I didn't think we would need to change the Golang files, but if they are involved, we should probably write an end-to-end test to verify they work.

Hey @calvinleungyk - would you be able to respond to the last comment?

Hi @ibzib, here's what I have so far:
I am not adding an ArtifactInformation of URL type in stager.py and am writing the remote file paths to EXTRA_PACKAGES_FILE = 'extra_packages.txt' in sdks/python/apache_beam/runners/portability/stager.py. This file is then read in installExtraPackages and pip tries to install the package directly, which will fail on private GCS bucket. If I generate an ArtifactInformation of URL type, the worker will eventually run extractStagingToPath on all ArtifactInformation and checks if the ArtifactInformation has a URNStagingTo role or if the type is URNFileArtifact, and both evaluate to False and the function will give an error.
I might be missing some place where the worker is using the artifact service to download artifacts as I'm not familiar with the worker code. If the above is inaccurate, would you be able to show me where the worker would attempt to fetch a URL artifact?
As for integration tests, I am running into credential issues which prevents the job from reaching Compute Engine Metadata server with error WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable onattempt 1 of 3. Reason: timed out,
WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Connecting anonymously.
...
Failed to start a local webserver listening on either port 8080
or port 8090. Please check your firewall settings and locally

The Gradle error is:
FAILURE: Build failed with an exception.

* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

extractStagingToPath

It looks like @ihji added those checks. He will know better what logic we need to change. I'm guessing these checks are simply overly strict. We may be missing adequate testing of different artifact types in materialize.go.
For tests, I recommend using a bucket in your own project at first. As the error message points out, you will need to set GOOGLE_APPLICATION_CREDENTIALS on your machine. You can do this by creating a service account and then setting GOOGLE_APPLICATION_CREDENTIALS to point to the key. Instructions here: https://cloud.google.com/docs/authentication/getting-started

I noticed there's also a DEFERRED artifact type, perhaps that could be of some use here?

Thanks @ibzib. Was successful in running the integration tests now.
@ihji would like to get some feedback from you on checks in extractStagingToPath function and how to support URL or DEFERRED type artifacts?
Thanks!

@ibzib @ihji @calvinleungyk - what's the plan for this PR?

Hi @pabloem, I am planning to take a stab at this and write a simple end to end test to test it. But I'm not familiar with the Golang part of the codebase and would take some time. So I was hoping @ihji would be able to give me some pointers regarding artifacts in materialize.go. If not, I will troubleshoot with an end to end test next week.

Is there any reason you want to download remote packages from the SDK harness? From the perspective of Dataflow runner, I personally don't see much benefit since downloading from third-party services (Azure, AWS, PyPI, Maven, etc.) every time the SDK harness boot-up seems vulnerable to network instability or third-party service failures. Dataflow job would fail when third-party services are unavailable even all GCP services are green.
The SDK harness won't download anything itself based on extra_packages.txt. All package files listed in extra_packages.txt should already exist in Docker container's staging location when pip installs them. To implement deferred remote package download, you need to 1) create URL artifact information for remote artifacts and add only local file names (matched with staging_to names from URL artifact information) in extra_packages.txt 2) make sure that URL artifact information is passed through the SDK harness without being materialized during job submission 3) extend materialize.go to support  non-GCS URL artifact information.
Also, please note that Dataflow uses some google-internal Python SDK harness boot-up codes. So this PR could not be merged (at least for a few months) until Dataflow fully migrate to the public Python SDK harness container.

@ihji We are using Dataflow and BEAM with mainly GCP services and GCS storage. The problem we are running into is that we could not upload remote packages to GCS to be used from Dataflow on Kubeflow Pipelines, and need to rebuild a custom image that contains the remote package every time, which works but is a hassle to our users. Since Dataflow needs to fetch staged artifacts either way, it seems logical to add support for that as well and leave network instability issues for the user; this feature was also supported previously on BEAM but removed at some point.
If you think supporting non-GCS artifacts would be a bigger issue, we could scope it down to only GCS to reduce reliance on third-party services. @ibzib and I thought that it would be useful to generalize it in previous discussions.
I have already created URL artifact information and populated with URL payload, but the blocker here is the support for that in materialize.go. The approach I'm using right now is to trace through how artifacts are materialized and it seems like I would have to add logic in extractStagingToPath so that it doesn't reject URL artifacts. Could you elaborate a bit more on "without being materialized during job submission"?
Also, do you have an issue or tracker for the public Python Dataflow SDK harness work?
Thanks for your help!

@ihji Specifically, to clarify what I mean by "scoping it down to GCS artifacts", would we still need to wait for Google to migrate to public Dataflow SDK harness? This would allow us to merge that part first and come back to the HTTP part and add support for third party services and endpoints.

Hi, just following up on this. Wondering if we are able to add and merge the GCS related changes first so that it's not blocked by Google's migration to public Dataflow SDK harness. We can work on a separate PR to add support for delaying all HTTP remote package downloads. cc @ihji @ibzib

I do not believe we should make this change. And this is not related to Google's private packages. The reasons are:

Staging packages creates a consistent set of artifacts throughout the lifetime of a pipeline. (Consider a long running streaming job, with autoscaling etc, different works might be started with days of gap). Downloading from a location other than staging location could result in different dependencies in different workers (e.g. a package getting an update in pypi).
If users want to do this they can do it by using a custom containers. Beam's custom container protocol (https://s.apache.org/beam-fn-api-container-contract) and allows for a completely custom container with support for changes like this (https://beam.apache.org/documentation/runtime/environments/)
It is also possible to achieve the results here even without using custom containers. (This is option is good for legacy Dataflow pipelines as well) by using a custom commands setup.py file (https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#nonpython). This file can execute any code at worker startup time including downloading and installation of arbitrary dependencies.

For 1., I would argue that it's the user's responsibility to ensure the pipeline is reading a consistent set of artifacts. If the user doesn't freeze package version in Pythons requirements.txt, they will get inconsistent libraries downloaded across regular Python job invocation. Even for the GCS staging location here, it's still overwritable and nothing much is done to ensure the artifacts are consistent across pipeline runs.
Custom containers could be a feasible solution and we are currently looking into it. The downside to this is users need to re-build a container every time they change or add dependencies/ files and this does not provide the best user experience (while being flexible). For the setup.py approach, users would need to learn how to write and structure the setup.py file before they can start testing with the pipeline, which increases the overhead and introduces friction for rapid experimentation.
For our use case, users compile a TFX pipeline (that uses Beam) on a local machine with extra_packages and then send that to a remote machine in a Kubeflow cluster. When the Kubeflow machine runs the pipeline, it only has the pipeline but not the extra_packages files. As extra_packages only support local paths, the job that is launched on remote Kubeflow machine fails. In Dataflow runner's case, GCS buckets are already used as staging locations so it doesn't seem such a big change to defer GCS downloads to Dataflow workers.
Alternatively, if we don't want to defer GCS downloads to Dataflow workers, we can use an approach similar to what was originally proposed in https://issues.apache.org/jira/browse/BEAM-11275, to support GCS path in stager.py so that the remote runner can download the package from a GCS path (not just local) and upload to staging bucket.
Compared to rebuilding containers multiple times or having our users learn how to write and structure a setup.py properly, this provides the convenience that matches the existing user experience, so ideally we can merge this change for support GCS paths somewhere.
Would be great if @aaltay and @ibzib could review this proposal. Thank you very much!

I still do not think this is the right solution. The reason is:
Ideally user's would be responsible for their dependencies being frozen but that is not the case, and that is not how user's are operating today. Even with frozen dependencies it is possible for sub-dependencies to change, or even dependencies change with the same version (as far as i know, pypi deos not prevent this). Additionally, with the GCS staging folder changing dependencies would require an active user action, with pypi it could chage without any user interaction. (More issues, pypi does not have SLOs as high as GCS, and Dataflow pipelines failing at startup because of pypi dependencies is common user issue.)
I understand the use you are describing. I would think that custom containers would actually result in a faster experience. Note that with custom containers that containers is built only once in the local machines, without that the same container will be built N times, one for each worker.
The suggestion to add support for remote package to the extra_package sounds good to me. I actually thought this is already the case (see: [1]). Is it not working? It seems like it does not support gs:// scheme but supports https://. You could add gs:// support there, or I believe you could get https url for files that exists on GCS and use that instead.
[1] https://github.com/apache/beam/blob/abfd4c662a6701feef078c36551c15bf4303eef4/sdks/python/apache_beam/runners/portability/stager.py#L577

Thanks for the response. That makes sense. I will go ahead and make the change to upload gs:// remote package to staging location, as in some cases custom containers work better, in others having GCS paths as deps work better.
You are right that it does not support gs:// scheme. for https://, it does not work well with private objects on GCS so I would prefer the gs:// support here.
Will update the code and ping for a review later. Thanks!

retest this please

@aaltay @ibzib I updated the PR and it's much shorter now. I did not add test because the _download_files method was mocked in all tests - let me know if you would like me to add new tests. In addition, I called Filesystems.copy() directly as the function performs a check via FileSystems.get_filesystem() and throws an exception if the filesystem is not supported by apache_beam.io.filesystems.

@aaltay @ibzib Ping

This change LGTM. Have you run a manual test with the new flag? I understand that mocking would make testing harder but it would be good to at least verify manually.

I updated the PR and also tested the code with the following command (private info redacted):
python3 sdks/python/apache_beam/examples/wordcount.py --input gs://dataflow-samples/shakespeare/kinglear.txt --output gs://<scratch-bucket>/counts --runner DataflowRunner --project <project> --region <region> --temp_location gs://<scratch-bucket>/tmp_beam --extra_package="gs://<gcs-bucket>/extra.whl" --sdk_location=container --no_use_public_ips --service_account_email=email.com --network=network --subnetwork=https://www.googleapis.com/compute/v1/projects/... --experiment=shuffle_mode=service

The job launched successfully with the following logs:
INFO:apache_beam.runners.portability.stager:Downloading extra package: gs://beam-dataflow-it/wheel/tfx_twitter.whl locally before staging
INFO:apache_beam.runners.portability.stager:Copied remote file from gs://beam-dataflow-it/wheel/tfx_twitter.whl to /var/folders/jl/3vwrt5kd6vg9vrhjpyy6b8dh0000gp/T/tmpm_bqr7q1/tmp9n_at3dp/tfx_twitter.whl.
...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://scratch-user.calvinl.dp.gcp.twttr.net/tmp_beam/beamapp-calvinl-1005095238-436424.1633427558.436689/tfx_twitter.whl in 0 seconds.

However, the job did not finish due to an internal GCP issue that leads to Error syncing pod  on Dataflow. If you'd like to see a finished job, I can add the logs once we resolve that internally.

I'm currently seeing
"Executing: /usr/local/bin/pip install --disable-pip-version-check --upgrade --force-reinstall --no-deps /var/opt/google/dataflow/local_deps-1.0.tar.gz"
"Processing /var/opt/google/dataflow/local_deps-1.0.tar.gz"
"gzip.BadGzipFile: Not a gzipped file (b'lo')"
"Failed to install packages: failed to install extra packages: failed to install extra package local_deps-1.0.tar.gz: exit status 2"

but it worked when I directly upload the dep as a local package.
Trying to install a simpler package to see if it's an issue with the file or the download mechanism.

@aaltay apologies, I meant I can confirm this code works on wordcount and for our internal use case and is ready to be merged! Sorry for the confusion

Perfect. We can merge once the test pass.
Do you mind adding any possible simple unit tests? It seems like we don't have much code coverage. This would be to prevent a future change from from breaking this flag.

@ibzib curious if there's any estimate as to what release this commit may be eligible to join into? (curious if it can be added to 2.34 if it's not too late?)

Thanks for helping me merge this! Is there an estimate of when 2.35.0 will be released?

The 2.35.0 release cut is scheduled for Nov 17, so a rough estimate for the release date is +5 weeks, around Dec 22.


docs: Clarify you need to upgrade from git for Zulip Cloud exports.

Looks great, merged, thanks @rishig!  Though possibly it's worth doing another pass on this to add numbered steps now that it's got a few.


switch from poetry to requirements.txt

This PR updates our Python dependency setup to use plain old requirements.txt instead of poetry, since some users have encountered issues with poetry.

PR with updates to the associated blog post: https://github.com/replicate/replicate-web/pull/2021


Enhance 'flooding' Property to Include Detailed Flood Risk Information

I propose enhancing the 'flooding' property in the PDTF specification to include more detailed flood risk information. The current structure allows for indicating whether a flood risk report has been prepared and whether any part of the property has been flooded. However, it lacks the ability to provide detailed flood risk assessments and related data.
"floodRisk": {
                      "title": "Flood Risk Information",
                      "type": "object",
                      "properties": {
                          "risk": {
                              "title": "Flood Risk Level",
                              "type": "string",
                              "enum": [
                                  "Very Low",
                                  "Low",
                                  "Moderate",
                                  "High",
                                  "Very High"
                              ]
                          }
                    }

Detailed flood risk reports form part of the searches under the Environmental content type.

The 'flood risk report' is part of the TA6 so has to be there and is arguably a duplication - once the enviro report returns we might choose to set that to "Yes" as well...

yes under environmentalRisk we do have riskResults but we want to provide atleast some sort of risk information under property details itself. may be we can remove details and just put flood risk level

Well we shouldn't invent a new structure for those results, since only the results as returned from the search provider will be provenanced. So we need to duplicate the entire flood report here.

yes that is correct

We did have a group conversation a while back about moving the search results into the body of the material info, and agreed not to. Should we revisit?

yes i think, we need to have some flood risk info in property details as most of the providers like sprift, street etc are providing flood risk as part of property details

We do have a schema working group catchup tomorrow at 12pm UK-time - @vivek656 let's raise this there shall we?

@vivek656 thinking about this some more last night and I'm coming round to the idea for sure. We should reconsider our 'searches are separate' thinking and move LLC, CON29R and DW in to the appropriate planning and water/drainage areas too. Let's confirm everyone's happy with this today and then I'd be happy to address this one 👍

@edmolyneux , me and others discussed this in biweekly meetings. and @edmolyneux will transfer(copy) results information from the searches section to material info. then we can add this data to the specific flooding risk report field.
@edmolyneux please, correct and add if I missed something

Will be addressed as part of #88


Possible bug with no-input-rename

Consider the following use-case:
any-component.html
<my-cmp [myInput]="'xx'"></my-cmp>

my-component.ts
@Input('myInput')
myInput: any;

The no-input-rename rule doesn't shows warning in this case above.
So.. I'm just wondering if it's intentional or no... because in fact it doesn't make sense to write myInput inside the parentheses: @Input('myInput') instead of @Input()... besides writing more, it won't affect anything at all.
PS: the same is valid for no-output-rename.

This rule seems to be intentional if we peek at the class test :

should succeed, when a directive input property rename is the same as the name of the property

Because the Style Guide indicates :

Two names for the same property (one private, one public) is inherently confusing.

In your case, it's the same name...
But, the style guide indicates also  :

You should use an alias when the directive name is also an input property, and the directive name doesn't describe the property.

It's a bit confusing ! Maybe, we can improve the style guide. @wardbell, what do you think about this ?
I think we can change Codelyzer in this direction. Are you agree  @mgechev ?

@wKoza maybe different error messages depending on the value passed to @Input.
Regarding the style guide, yes, probably we can make the explanation more complete.

I think this also warrants taking another look at https://github.com/mgechev/codelyzer/issues/224.
The "Angular Fundamentals" specifically recommends aliasing an input variable to the directive selector
(https://angular.io/guide/attribute-directives#binding-to-an-input-property), so I really think Codelyzer should recognize this special case and not flag it as an error.
(I am aware that it can be selectively disabled, but it feels wrong to actively have to fight the linter to write idiomatic code).

Lets move it to the next milestone.


New Rule Proposal for jsx-curly-spacing

I would like to be able to do the following:
"jsx-curly-spacing": [2, "always", {"alternative": true}]

When "always" is used and alternative is true, following patterns are not warnings:
<Hello name={ firstname } />;
<Hello name={{ firstname: 'John', lastname: 'Doe' }} />;
<Hello name={
  firstname
} />;

the following patterns are considered warnings:
<Hello name={firstname} />;
<Hello name={ firstname} />;
<Hello name={firstname } />;
<Hello name={ {firstname: 'John', lastname: 'Doe'} } />;

If I understand correctly you would like something similar to the objectsInObjects option in the ESLint object-curly-spacing rule. I think that's something we can do.

Awesome! Thank you.

I think this is similar to #211.

I get Configuration for rule "react/jsx-curly-spacing" is invalid: Value "data["1"].alternative" has additional properties when trying to do "alternative": true as a third argument

@casesandberg yes, alternative is not a valid option since this proposal is not implemented yet.

Ah, got it @yannickcr, I will see if I can get it in tonight!


feat(FN-2311): filter only premium payments

Introduction :pencil2:
Applying the existing 'Filter by facility ID' filter on the premium payments tab (TFM UI Bank reports page) applies this filter to all payment groups across the other tabs (for example, the payment details tab rows are filtered by this, despite it being a different tab) - see screenshot. We want to fix this filtering such that it is only applied to the premium payments tab. Additionally, this ticket will be introducing additional filters on other tabs (under a subsequent PR) so we want to group the filters by tab.

Resolution :heavy_check_mark:

Existing premium payments tab filter only applied to that tab.

Instead of returning a single array of fee record payment groups, both an unfiltered array and a filtered array for only the premium payments tab is returned.
Updated tests to reflect this change

Filters now passed around as a group for a given tab.

Miscellaneous :heavy_plus_sign:
N/A

Apologies, the TFM UI unit tests were failing as a parameter name was incorrect - just pushed a commit to fix that. Would you mind reapproving please @ttbarnes if you're happy, thanks very much


Updated validations.py python script.

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Updated validations.py python script.

Updated validations.py python script.


TypeError: PSD.fromFile is not a function

version: 3.3.0
bug:

And you are importing it this way: import PSD from 'psd' ?

I just ran into this issue as well using npm to install latest of psd.
@Gasegamer - importing index.js worked for me, thanks!

Have a pr that may resolve this so we don't have to do that wonky workaround, I'll do a bit of testing and work on a release.
PR: https://github.com/meltingice/psd.js/pull/244

PSD = require('./node_modules/psd/index');

It worked for me, thank you

This should be fixed in the latest release 3.4.0.

This should be fixed in the latest release 3.4.0.

good work


bond_core: 1.8.2-0 in 'melodic/distribution.yaml' [bloom]

Increasing version of package(s) in repository bond_core to 1.8.2-0:

upstream repository: https://github.com/ros/bond_core.git
release repository: https://github.com/ros-gbp/bond_core-release.git
distro file: melodic/distribution.yaml
bloom version: 0.6.4
previous version for package: 1.8.1-0

bond

No changes

bond_core

No changes

bondcpp
* uuid dependency fixup (#36 <https://github.com/ros/bond_core/issues/36>)
  * dont export uuid dependency as this isnt anywhere in the public api
  * fixx uuid dependency in test_bond as well
* Contributors: Mikael Arguedas

bondpy

No changes

smclib

No changes

holding to sync with other ros_base package's releases


Pipfile should explicitly specify an encoding

Spun off from #1842.
Currently Pipenv does not handle non-ASCII characters very well. Pipfile is parsed with the platform’s default encoding; this becomes a problem for cross-platform projects if there are non-ASCII paths in the project, especially for the scripts section, but also for local dependencies’ path value.
A straightforward solution would be to strictly require Pipfiles to be encoded with UTF-8. This is in line with the best practice of the Python community in general, including core development. Pipenv would then use io.open instead of open to read Pipfiles instead (in project.py), so contents, including paths, are always read as text in the correct encoding.

I think this makes a lot of sense, does anything break if we implement it right now?

Yes, if you already have a platform-specific encoded Pipfile with non-ASCII stuff in it. But this is easily amendable by re-saving the Pipfile with UTF-8.

How can we gracefully warn people about that when we encounter it?

The parsing would almost always fail with UnicodeDecodeError if the encoding is not UTF-8, so it would probably be enough to catch that and emit a prettier error.
I doubt that would be necessary though. This would happen mainly on 1. Windows 2. POSIX systems with non-UTF-8 encoding. The latter is extremely unconventional these days (and I believe is not supported anyway). The former is probably next to non-existent, based on how many fundamental issues brettcannon (avoiding an @ here) was able to file when he adopted Pipenv through VS Code.

Can we fall back to their default system encoding if we handle that exception?

@jtratner Yeah it needs careful testing, but it can be pulled off. Many projects already did.

@uranusjr @jtratner relevant link from Victor Stinner from yesterday regarding 3.7 which will by default open things in utf-8 mode: https://vstinner.github.io/python37-new-utf8-mode.html

Yes, encoding shall be agreed and not dependent on current default encoding. Otherwise we will repeat errors as in distutils or in pbr package.
For Python 2.x compatibility, this ConfigParser snippet from PyMoTW could help.

@ncoghlan if you have a sec and a general comment on this one also I’d be super interested

I'd missed that we weren't already enforcing UTF-8 everywhere - we definitely should be.
As @uranusjr notes, that's straightforward for new files and for ASCII-only files, which means the main cases to handle are going to be existing locale encoded files on *nix systems and code page encoding files on Windows.
We're helped here by the fact that PyPI itself restricts package names to ASCII-only identifiers, so non-ASCII characters can only appear in comments, and in non-PyPI package names and URLs.
I figure that just fixing the encoding (for files pipenv was about to edit anyway) is going to be easier than remembering to preserve the original encoding when writing them back, so I think transparently resolving the problem will be straightforward enough to be worth doing.
(@techalchemy Regarding the UTF-8 mode change in 3.7: that's only the default in the POSIX or C locale, although I think it may be the default on Windows in 3.6+)

Thanks for the input @ncoghlan I know you have a lot of projects to juggle so I appreciate it. I agree we are handling pretty much universally ascii characters and before yesterday I’m not sure we actually preserved all comments to begin with. So I agree in light of that it would seem worth pursuing (the risk seems low)

This reminds me, I also discovered we’re also writing CR/LF line endings on Windows. This will need to be normalised as well.


Update scenario-desktop-overview.md

Corrected grammar for incorrect use of neither/nor after a negative (won't). Also corrected capitalization of IoT (was "iOT")

@peterfoot : Thanks for your contribution! The author(s) have been notified to review your proposed change.

confirmed corrections


Add support for custom output processor

The goal is allow users to create any output generator, and use any template engine or JS code instead of coupling to Handlebars.
The idea is to allow a custom processor from a JS file (as default export of a function), for example with JS template literals:
export default function(contextType, compilationContext) {
    return `${compilationContext.name} = {...};`;
}

and use to with the CLI:
gql-gen --processor ./my-custom.js

And the current Handlebars will be just another function that exposes Handlebar's compile function.

Done in: https://github.com/dotansimha/graphql-code-generator/pull/268

Done in 0.9.0 :)


[nodejs] mount-require is only reloading the main source file, not any other required files

Moved from https://github.com/adobe/wskdebug/issues/45
With an action project structure like this:
package.json
core.js
lib/
  helper.js

where core.js does require('./lib/helper.js') and a wskdebug invocation such as
wskdebug core core.js -l

it will see all file changes, but it won't actually reload anything other than the main file (core.js). This is because here
https://github.com/adobe/wskdebug/blob/master/src/kinds/nodejs/mount-require.js#L37
we are only clearing the require cache for the main file. It should also delete any subpaths or all deps (maybe someone did an npm install in between and node_modules should also be reloaded).

Links:

https://github.com/sindresorhus/clear-module
https://stackoverflow.com/questions/23685930/clearing-require-cache/23686122#23686122


Make reading of new types of data files consistent

Currently, I'm reading new types of headers by making the program conform to the type of header - I should probably change it, so that the header is reformatted into a standard form that is then always used by the program

Files are now being read and the columns are being renamed to match the key-names in the self.read_options dictionary. New column names to handle can be added to the relevant (key, value)-pair.
Added checks for
FileNotFoundError

if QFileDialog.getOpenFileName() does not return a filename
UnicodeDecodeError
if the file is not a text file
FileFormatError
custom error to handle if the file does not have a format that can be read into a pandas DataFrame
AssertionError
asserting that relevant file names should not appear more than once
AND
asserting that magnetization and susceptibility do not appear simultaneously


Issue #130: Properties with dashes in their names

Everything appears to work as it did previously, but this modification now handles the use case where the declarations variable could have a dash in it (EX: "value:background-color"). Additionally the declarations variable is passed into the function, to ensure it's defined.
This pull request includes the type conversion that happened automatically by creating a function using the function constructor.

Ah another case I don't think I have covered: Having an object passed in splits it irregularly.  Playing around with trying to keep the type conversion behind the scenes without an error being thrown.


[hi] Add content/hi/docs/tutorial/kubernetes-basics/_index.html

Hey folks, this is a PR for hi localization of the file: content/hi/docs/tutorial/kubernetes-basics/_index.html
PR is linked under this issue #29353
/language hi

Please note that the directory name is tutorials and not tutorial.

Please note that the directory name is tutorials and not tutorial. Change the name of directory to avoid conflicts in future : ) Thanks

Sorry, I made the changes, should I made the PR ready, I mean to move from draft mode.

@rajatgupta24 No, keep the PR in draft mode for now.
Also, I can't see any files in File changed section.

@rajatgupta24 No, keep the PR in draft mode for now. We will open PRs for review once we get a separate branch.
Also, I can't see any files in Files changed section.

@anubha-v-ardhan, check now

/assign @mittalyashu

/retitle [hi] Add content/hi/docs/tutorial/kubernetes-basics/_index.html

@rajatgupta24 You can change the base branch to dev-1.22-hi.1

/assign


chore(package): update chai to version 4.0.2

Codecov Report

Merging #47 into master will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff          @@
##           master   #47   +/-   ##
====================================
  Coverage      92%   92%
====================================
  Files           1     1
  Lines          25    25
====================================
  Hits           23    23
  Misses          2     2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9201c58...9145bb2. Read the comment docs.


Clicking the own should not bring you to the loading screen

Clicking the owl should not currently do anything.

Remove the hover and highlight and pointer. Make it just a static image.


Simple update leaving us with broken project

Doing a simple "apm update" is starting with uninstalling packages and removing them from project.apm without any plausiblity check it seems.
Only on install as a last step, I get hit with e.g.
CONFLICT :: fatal error : found [1] conflicting packages
CONFLICT :: com.distriqt.Core
CONFLICT :: ├── 7.0.1 required by<EMAIL_ADDRESS>CONFLICT :: └── 6.4.8 required by<EMAIL_ADDRESS>
Causing me half an hour of work reverting everything manually, reinstalling it, double checking the versions and testing all functionalities.
Obviously that should not happen in any circumstance.

In fact this conflict bricked the entire project. It is impossible for us to revert back, effectively preventing us from app updates.
Even reverting back everything is internally still referencing libs we have no control over:
CONFLICT :: fatal error : found [1] conflicting packages
CONFLICT :: com.distriqt.Core
CONFLICT :: ├── 7.0.1 required by<EMAIL_ADDRESS>CONFLICT :: └── 6.4.8 required by<EMAIL_ADDRESS>::

All of our extensions are getting updated. Please log an issue in the relevant extension and we'll prioritise the update.

How does this resolve the broken update process? It should never come to bricking a project because of a random conflict. That's literally what a packet manager has to guarantee. We can't even revert back to old versions because the deps are pointing to newer versions that are unusable (like .AppSet) - how does this even happen? It completely defeats the reason behind a packet manager.
And as google currently demands updates for e.g. unityAds sdk, we are dead in the water.

Hi, Thanks for the suggestions and please understand this is a project developed by me in my spare time so is very much in it's infancy.
The AppSet dependency was an issue I accidentally introduced and broke that package. It is resolved as I updated that package and it shouldn't block installing using an existing project file now.
distriqt is very busy updating all extensions for Xcode 14 support, this has meant that the latest ANEs are not compatible with older versions so if you try to update certain extensions at the moment you will run into a conflict, similar to the above.


Found link https://files.pythonhosted.org/packages/4b/ab/3e42d1f0227e3a23d02338506e24007a1d9c21bda028735efb19285578f9/future-0.14.0.tar.gz#sha256=af25a0ee915d8d8720d8fe8b4516e55ac27f354d5d57af3c900757f4b6ccca8c (from https://pypi.org/simple/future/), version: 0.14.0
    Found link https://files.pythonhosted.org/packages/dc/a6/57718937bc25e602367d82ff86a6689b677f18e29e30ff9bab58cc17f5dc/future-0.14.1.tar.gz#sha256=854d893d57b894193169c65661884fd318d6e106ae84c1fb9f30604d89ee9de1 (from https://pypi.org/simple/future/), version: 0.14.1
    Found link https://files.pythonhosted.org/packages/d6/c1/3e8cee376151bd70744e7f1290212f887333323f260da22eb834c709d236/future-0.14.2.tar.gz#sha256=1af14fadd7779520345b4376ca24c4ff16fe9e59cbfb453dfe7a3e44fe76a1f7 (from https://pypi.org/simple/future/), version: 0.14.2
    Found link https://files.pythonhosted.org/packages/83/80/8ef3a11a15f8eaafafa0937b20c1b3f73527e69ab6b3fa1cf94a5a96aabb/future-0.14.3.tar.gz#sha256=62857d51881d97dd5492b9295b9f51d92108a52a4c88e2c40054c1d3e5995be9 (from https://pypi.org/simple/future/), version: 0.14.3
    Found link https://files.pythonhosted.org/packages/7c/eb/1d7403c6d187ec097394685b0ea8a69faadaeb63f222d6c9b85ae165f915/future-0.15.0.tar.gz#sha256=7f8ad46e287ac32e3dae78be41098c83d690b364142e0a5f11958c2b549420b0 (from https://pypi.org/simple/future/), version: 0.15.0
    Found link https://files.pythonhosted.org/packages/41/b7/b646f0140a752ea87bb0b4c3a4af6a1e6e5fa676b9b1314a702368b75d4b/future-0.15.1.tar.gz#sha256=d61405f32beb357eeef6e32bbfe4c0e463cac658f59601a5ddeabf87bb8c46c1 (from https://pypi.org/simple/future/), version: 0.15.1
    Found link https://files.pythonhosted.org/packages/5a/f4/99abde815842bc6e97d5a7806ad51236630da14ca2f3b1fce94c0bb94d3d/future-0.15.2.tar.gz#sha256=3d3b193f20ca62ba7d8782589922878820d0a023b885882deec830adbf639b97 (from https://pypi.org/simple/future/), version: 0.15.2
    Found link https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz#sha256=e39ced1ab767b5936646cedba8bcce582398233d6a627067d4c6a454c90cfedb (from https://pypi.org/simple/future/), version: 0.16.0
    Found link https://files.pythonhosted.org/packages/85/aa/ba2e24dcb889d7e98733f87515d80b3512418b80ba79d82d2ddcd43fadf3/future-0.17.0.tar.gz#sha256=eb6d4df04f1fb538c99f69c9a28b255d1ee4e825d479b9c62fc38c0cf38065a4 (from https://pypi.org/simple/future/) (requires-python:>=2.6, !=3.0.*, !=3.1.*, !=3.2.*), version: 0.17.0
    Found link https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz#sha256=67045236dcfd6816dc439556d009594abf643e5eb48992e36beac09c2ca659b8 (from https://pypi.org/simple/future/) (requires-python:>=2.6, !=3.0.*, !=3.1.*, !=3.2.*), version: 0.17.1
  Using version 0.17.1 (newest of versions: 0.0.1, 0.0.2, 0.0.3, 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.17.0, 0.17.1)
  Using cached wheel link: file:///root/.cache/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e/future-0.17.1-cp27-none-any.whl
  Added future from file:///root/.cache/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e/future-0.17.1-cp27-none-any.whl to build tracker '/tmp/pip-req-tracker-GR4fWV'
  Removed future from file:///root/.cache/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e/future-0.17.1-cp27-none-any.whl from build tracker '/tmp/pip-req-tracker-GR4fWV'
Installing collected packages: future

  changing mode of /root/.local/share/virtualenvs/test_future_27-MltuwF9Z/bin/pasteurize to 755
  changing mode of /root/.local/share/virtualenvs/test_future_27-MltuwF9Z/bin/futurize to 755
Successfully installed future-0.17.1
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-GR4fWV'
(test_future_27) [root@localhost test_future_27]# python
Python 2.7.15 (default, Oct 24 2018, 10:29:31)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from future import standard_library
>>> standard_library.install_aliases()
>>> import configparser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named configparser
>>> exit()

Our current workaround is to pip install configparser as well, rather than using the older version of future.

I'm unable to reproduce -- can you try to work out a root cause?

@jmadler unsure whether the "you" refers to me (as OP), but anyway: I cannot reproduce the problem anymore, it went away for me with the 0.17.1 release.

Awesome, thanks :)