Does the order of points in a published point cloud matter for local SLAM?

I assigned this issue to cartographer_ros since I am using it, but it might also be an issue for cartographer. I am testing the effect of (spatially lossy) compressions on local SLAM. The dataset I am using is KITTI (2D SLAM, using only cropped velodyne data, no IMU, one whole scan sent in one message).
I am testing a compressed and decompressed dataset with maximum deviation of 1cm, which should have (almost) no effect on local SLAM, assuming the voxel grid filter used is 15cm. Additionally, the compression and decompression modifies the order of points. I did some tests, which resulted in the following trajectories:

So the order of points seems to be the only thing that really matters in that case (correct me if I am wrong). Unfortunately, I was not able to find the reason for that in cartographer's code (I checked  CeresScanMatcher2D::Match as well as the cost function, but did not find anything). In what part of the code could the order of the points in the point cloud matter?
Here are my configuration files:
kit_2d.lua:
https://github.com/UserUsingGit/cartographer_ros/blob/master/cartographer_ros/configuration_files/kit_2d.lua
demo_kit_2d.rviz
https://github.com/UserUsingGit/cartographer_ros/blob/master/cartographer_ros/configuration_files/demo_kit_2d.rviz
kit_2d.launch
https://github.com/UserUsingGit/cartographer_ros/blob/master/cartographer_ros/launch/kit_2d.launch
demo_kit_2d.launch
https://github.com/UserUsingGit/cartographer_ros/blob/master/cartographer_ros/launch/demo_kit_2d.launch
kit_2d.urdf
https://github.com/UserUsingGit/cartographer_ros/blob/master/cartographer_ros/urdf/kit_2d.urdf
rosbag_validate gives the following output:
cartographer_rosbag_validate -bag_filename /home/user/data/kitti_2011_10_03_drive_0027/bagfiles/kitti_0027_0_000001m.bag
E0729 12:30:20.301599 22724 rosbag_validate_main.cc:389] IMU data (frame_id: "imu_link") has a large gap, largest is 0.120001 s, recommended is [0.0005, 0.005] s with no jitter.
I0729 12:30:20.301928 22724 rosbag_validate_main.cc:398] Time delta histogram for consecutive messages on topic "/kitti/oxts/imu" (frame_id: "imu_link"):
Count: 4543  Min: 0.079936  Max: 0.120001  Mean: 0.103640
[0.079936, 0.083943)	                    	Count: 1 (0.022012%)	Total: 1 (0.022012%)
[0.083943, 0.087949)	                    	Count: 0 (0.000000%)	Total: 1 (0.022012%)
[0.087949, 0.091956)	                    	Count: 5 (0.110059%)	Total: 6 (0.132071%)
[0.091956, 0.095962)	                    	Count: 0 (0.000000%)	Total: 6 (0.132071%)
[0.095962, 0.099969)	                ####	Count: 980 (21.571650%)	Total: 986 (21.703720%)
[0.099969, 0.103975)	            ########	Count: 1883 (41.448383%)	Total: 2869 (63.152103%)
[0.103975, 0.107982)	                    	Count: 63 (1.386749%)	Total: 2932 (64.538849%)
[0.107982, 0.111988)	             #######	Count: 1610 (35.439137%)	Total: 4542 (99.977989%)
[0.111988, 0.115995)	                    	Count: 0 (0.000000%)	Total: 4542 (99.977989%)
[0.115995, 0.120001]	                    	Count: 1 (0.022012%)	Total: 4543 (100.000000%)
E0729 12:30:20.302032 22724 rosbag_validate_main.cc:382] Point data (frame_id: "velo_link") has a large gap, largest is 0.104733 s, recommended is [0.0005, 0.05] s with no jitter.
I0729 12:30:20.302076 22724 rosbag_validate_main.cc:398] Time delta histogram for consecutive messages on topic "/kitti/velo/pointcloud" (frame_id: "velo_link"):
Count: 4543  Min: 0.102937  Max: 0.104733  Mean: 0.103652
[0.102937, 0.103117)	                    	Count: 4 (0.088048%)	Total: 4 (0.088048%)
[0.103117, 0.103296)	                    	Count: 30 (0.660357%)	Total: 34 (0.748404%)
[0.103296, 0.103476)	                   #	Count: 203 (4.468413%)	Total: 237 (5.216817%)
[0.103476, 0.103655)	          ##########	Count: 2225 (48.976448%)	Total: 2462 (54.193264%)
[0.103655, 0.103835)	            ########	Count: 1826 (40.193703%)	Total: 4288 (94.386971%)
[0.103835, 0.104015)	                   #	Count: 185 (4.072199%)	Total: 4473 (98.459167%)
[0.104015, 0.104194)	                    	Count: 64 (1.408761%)	Total: 4537 (99.867928%)
[0.104194, 0.104374)	                    	Count: 5 (0.110059%)	Total: 4542 (99.977989%)
[0.104374, 0.104553)	                    	Count: 0 (0.000000%)	Total: 4542 (99.977989%)
[0.104553, 0.104733]	                    	Count: 1 (0.022012%)	Total: 4543 (100.000000%)
If necessary, I will provide the bagfiles too, please let me know.
Any help/guess is appreciated.
Thank you very much in advance!

To be sure it is the order of points that affected local SLAM, I took the original Velodyne data, randomly shuffled the points in each frame and wrote them back to the binary file. Then I converted the respective data to bagfiles. I made sure the points keep the same types and accuracy as before. The results confirm the results from above.

I was not able to find the reason for this yet.

Cartographer expects ordered input. I can't help with your use-case, so you might need to check yourself what happens in detail if you are interested.


Flag sessions as deleted instead of force-deleting

Leave force-deletions to session refresh

Instead, just remove them from Redis and let the refresh take care of it.


Type hint in function in interface

When i wanted to type hint in function built-in php class then when all compiled fine, but it's looking for class in current namescape instead of built-in php class. There is no problem for the same class in another class when type hint. For example ReflectionClass etc.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/31881857-type-hint-in-function-in-interface?utm_campaign=plugin&utm_content=tracker%2F280146&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F280146&utm_medium=issues&utm_source=github).

I had to do fullclean to make it working cuz i forget about uses.


Can't register listeners inside onMounted (DOMContentLoaded / window.load events)

Environment
Nuxt 3.7.3 as well as 3.5 (other versions not tested)
Reproduction
https://stackblitz.com/edit/github-aeewhc?file=pages%2Findex.vue
or paste this in your page.vue:
<script setup>
onMounted(() => {
  window.addEventListener('load', (event) => {
    console.log('page is fully loaded');
  });
});
</script>

Describe the bug
It appears that when we use pages catalog we don't have access to load event listeners inside onmounted, like DOMContentLoaded. Scroll event works fine.
When I remove pages folder and stick to app.vue only, event listener works as expected and fires when page is loaded.
Additional context
No response
Logs
No response

Hi @mkierdev I don't think it is nuxt issue as I can reproduce it with just Vue. I can assume that browsers fires load event before app is fully mounted. Check if using beforeMount method will fit your needs: https://vuejs.org/api/options-lifecycle.html#beforemount
Vue reproduction link: https://stackblitz.com/edit/vue-sds8s8?file=src%2FApp.vue

In my case, onNuxtReady() was enough to fit my needs.
https://nuxt.com/docs/api/utils/on-nuxt-ready#onnuxtready


Database left in an incorect state if interrupting an ongoing deployment

An interrupted deployment keeps its status as RUNNING.
Is it a normal behavior? How to deal with this case as nothing failed but a few operations already succeeded?

It seems normal to left the database "as is" in this case. What we should look for is how to recover from this state (maybe using a resume?).

To my mind, if we have resume, it is naturally to have a pause command. In the context of TDP, it would be very useful.
Deployment process could be interrupted by the user in 2 manners:

sending a temination signal with CTRL+C or clossing the parrent process
running a command, for example tdp pause command or via tdp-server in the future.

Having pause, we will need another status PAUSED for deployments and operations. That would complement this current issue with a better usage.


Right-click to remove terrain

Rather than ray-based removal, create a "cube cursor"; any triangles partially inside the cursor will be removed when the player left-clicks.

Remove a voxel and any adjacent edges, redraw any terrain blocks they touch, and send updates to the clients.

https://github.com/bfops/playform/tree/remove-voxel

Right-click to remove voxels 5584598bef1ea71e5cb99bcef52d59552b47ee61


floating_action_button_theme missing from Theme

Duplicate Check

[X] I have searched the opened issues and there are no duplicates

Describe the bug
When trying to implement a FloatingActionButtonTheme and add to the ft.Theme class there is no property to override.
Code sample
Code
class MyFloatingActionButtonTheme(ft.FloatingActionButtonTheme):
    def __init__(self):
        super().__init__(
            bgcolor='#FFFFFF',
        )

class MyTheme(ft.Theme):
    def __init__(self):
        super().__init__(
            floating_action_button_theme=MyFloatingActionButtonTheme(),
        )

To reproduce
Try to initialize the above theme.
page.theme = mypackage.MyTheme()

TypeError: Theme.__init__() got an unexpected keyword argument 'floating_action_button_theme'
Expected behavior
FAB theme should be overridden.
Screenshots / Videos

Captures
[Upload media here]

Operating System
macOS
Operating system details
MacOS 14.5
Flet version
0.22.0
Regression
No, it isn't
Suggestions

https://github.com/flet-dev/flet/pull/3771

Logs
Logs
[Paste your logs here]

Additional details
No response

Workaround for me is to assign it to theme directly:
page.theme.floating_action_button_theme = MyFloatingActionButtonTheme()


Crash on Mac after model is loaded, can't find how to enable logs

Hi, thank you so much for your work!
I need to eventually run this on a GPU but for testing I'm running on my mac and for some reason I get to the generator loop point and then the process crashes with
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
This is the whole output:
2023-06-16 18:21:53.727 | INFO     | faster_whisper.transcribe:transcribe:243 | Processing audio with duration 02:54.010 | {}
Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
/Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

I'm not sure what I did wrong, I tried to add CT2_VERBOSE=1 and I pasted the logging config but those are all the logs I'm able to get.
Not sure what else to check since the logs don't say much.
Here's my relevant implementation snippet:
    import logging
    logging.basicConfig()
    logging.getLogger("faster_whisper").setLevel(logging.DEBUG)
    from faster_whisper import WhisperModel

    model = WhisperModel('tiny', device="cpu", cpu_threads=8, compute_type="float32")
    props.update({
      "audio": source_file,
      "beam_size": 5,
      "vad_filter": False,
      "vad_parameters": dict(min_silence_duration_ms=500),
      "initial_prompt": initial_prompt,
    })

    if language != 'autodetect':
      init_language = TO_LANGUAGE_CODE[language.lower()] if len(language) > 2 else language
      detected_language = init_language
      props["language"] = init_language
      # this will make it run faster cause we know the language
      segments, info = model.transcribe(**props)
    else:
      segments, info = model.transcribe(**props)
      detected_language = info.language

    try:
      segments = list(segments)  # The transcription will actually run here.
    except Exception as e:
      log.info(f"Could not transcribe file: {e}")
      raise
    output = {
      "language": detected_language, "segments": segments
    }

Would really appreciate a hint here, I really want to speed up my whisper inference 🙏

Hi,
What’s the model of your Mac and what macOS version are you running?

Hey
MacBook Pro with M1

It seems you installed the wrong binary for CTranslate2. There is a log message from Intel MKL but this library is not used in the binary targeting Mac M1.
How did you install this project?

It seems you installed the wrong binary for CTranslate2. There is a log message from Intel MKL but this library is not used in the binary targeting Mac M1.
How did you install this project?

Just literally "pip install faster-whisper" in a conda, and it picked up CTranslate2.
I will try to install it directly and see if this helps?

Yes, try a clean installation and post the full installation logs.

Yes, try a clean installation and post the full installation logs.

I tried to reinstall and it used a cached version with this exact wheel:
pip uninstall ctranslate2
Alias tip: pipun ctranslate2
Found existing installation: ctranslate2 3.16.0
Uninstalling ctranslate2-3.16.0:
  Would remove:
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-fairseq-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-marian-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-openai-gpt2-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-opennmt-py-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-opennmt-tf-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-opus-mt-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/bin/ct2-transformers-converter
    /Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/site-packages/ctranslate2-3.16.0.dist-info/*
    /Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/site-packages/ctranslate2/*
Proceed (Y/n)? y
  Successfully uninstalled ctranslate2-3.16.0
pip install ctranslate2
Alias tip: pipi ctranslate2
Collecting ctranslate2
  Using cached ctranslate2-3.16.0-cp39-cp39-macosx_10_9_x86_64.whl (14.4 MB)
Requirement already satisfied: pyyaml<7,>=5.3 in /Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/site-packages (from ctranslate2) (6.0)
Requirement already satisfied: numpy in /Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/site-packages (from ctranslate2) (1.23.5)
Installing collected packages: ctranslate2
Successfully installed ctranslate2-3.16.0
1.91s user 0.18s system 87% cpu 2.398s total

The I uninstalled again and installed w / --no-cache-dir and it installed the same version and the same wheel
pip install --no-cache-dir ctranslate2
Alias tip: pipi --no-cache-dir ctranslate2
Collecting ctranslate2
  Downloading ctranslate2-3.16.0-cp39-cp39-macosx_10_9_x86_64.whl (14.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.4/14.4 MB 6.1 MB/s eta 0:00:00
Requirement already satisfied: numpy in /Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/site-packages (from ctranslate2) (1.23.5)
Requirement already satisfied: pyyaml<7,>=5.3 in /Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/site-packages (from ctranslate2) (6.0)
Installing collected packages: ctranslate2
Successfully installed ctranslate2-3.16.0

And alas, the same result:
2023-06-19 13:28:50.634 | INFO     | faster_whisper.transcribe:transcribe:243 | Processing audio with duration 02:54.010 | {}
Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
/Users/altryne/opt/miniconda3/envs/detadownloader/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 9 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Figured I'd go ahead and try this in a complete fresh Conda env and it... showed more of the error, maybe be helpful to pinpoint?
2023-06-19 13:54:56.151 | INFO     | faster_whisper.transcribe:transcribe:243 | Processing audio with duration 02:54.010 | {}
Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

The issue is that your environment downloads the x86_64 wheel while it should download the arm64 wheel.
I think you installed the x86 version of Miniconda3. Can you check that?

The issue is that your environment downloads the x86_64 wheel while it should download the arm64 wheel.
I think you installed the x86 version of Miniconda3. Can you check that?

Wasn't sure how to check that, but I just setup a venv and looks like you're right?
without conda, I'm getting this wheel:
Downloading ctranslate2-3.16.0-cp311-cp311-macosx_11_0_arm64.whl
And running via the venv seems to work! (getting another error but will post a separate issue)
🙏
Hope this helps someone who has the same issue ⬆
Make sure your environment picks up the right ctranslate wheel


Add issue template to divert babili issue to the correct repo

It seems that people are reporting what is probably babel issue here, when it is better done at the babili repo.

❤ - Thank you!!!


CMS-580: Fix bug that prevents sync script to get all the data.

Jira Ticket
CMS-580
Description
Followed the steps described in the ticket to test.
syncData() can run as many times as you want
oneTimeDataImport() should only run one time.
Note that campgrounds are not created here given that the names of the subareas in Strapi are not consistent enough to assign campsite groupings to campgrounds.
I created a separate script to help with the process, but it will probably need to be verified manually.

After the discussion this morning, we still want to confirm with Rob but I think we can continue as planned with bringing over the campsite groupings, and make a note of any changes to Strapi that would make this process more solid (adding ID fields, etc).

[ ]  Pull in campsite groupings, (discuss & confirm w/ Rob or Brett?)

As discussed, let's also add commands to the scripts in package.json so we can run these things easily in every environment.

[ ]  Add npm run command(s)

I imported the data by running syncData and oneTimeDataImport once each, but it looks like some FK associations might be broken, I tried to add a date range to Buttle Lake Campground under "Strathcona Park 2024 season dates" and I got an error from the backend:

"insert or update on table "SeasonChangeLogs" violates foreign key constraint "SeasonChangeLogs_userId_fkey""

[ ]  Check on foreign key errors

After the discussion this morning, we still want to confirm with Rob but I think we can continue as planned with bringing over the campsite groupings, and make a note of any changes to Strapi that would make this process more solid (adding ID fields, etc).

[ ]   Pull in campsite groupings, (discuss & confirm w/ Rob or Brett?)

As discussed, let's also add commands to the scripts in package.json so we can run these things easily in every environment.

[ ]   Add npm run command(s)

I imported the data by running syncData and oneTimeDataImport once each, but it looks like some FK associations might be broken, I tried to add a date range to Buttle Lake Campground under "Strathcona Park 2024 season dates" and I got an error from the backend:

"insert or update on table "SeasonChangeLogs" violates foreign key constraint "SeasonChangeLogs_userId_fkey""

[ ]   Check on foreign key errors

I think the campgrounds should be a separate ticket since it's not part of the syncing process. It's just an extra step we're doing to keep the data cleaner, but it's not based on specific rules. More on a guess that should be documented somewhere else.
For the error you're getting, I think just just need to create a user in the admin dashboard. Since we are not getting the user from the request in the backend yet, right now it's just setting the userId of SeasonChangeLog to 1. To properly assign the userId to the model, we'd need to wait for the ticket in which we add auth to the API.

Nice, you're right about the missing user record. That's gonna happen in the 10-point big ticket so hopefully that's not too far away. Do you want to create the ticket for the campsite grouping and include all of the things we're not certain about in the description? We can create it in the backlog but feel free to bring it into the sprint if we get to it.

Gonna merge this one and try to deploy it on dev and test before the end of the day 🫡 🚢

FYI, I just had to push a fix because the import path in the oneTimeDataImport script didn't work. We have to include the file extension for ES modules to work ✅


Content Language

Description of Problem or Question
We've had an I interesting issue raised on the BGG forums.
A polish player owns the English base game. When he plays he struggles to match the Polish names of the content in app to the English titles he owns, because the tile orders and names are in English.
It's made me think if there is an elegant way to address this.
Would it be possible to have a language 'override' on each of the expansions in content (and add one for the base game).
Here's my example.
An English speaking gamer owns MoM base, however he invites hit friends over, one if which owns BtT in French. They decide to play in English, but all of the content for BtT is listed in the app as English, therefore the tile names need to be translated by the players as they play.
Having an option would mean that players could override the default to leave MoM base as English but set BtT to French.  It wouldn't happen to your average person as it's a specific setting they would have to set when  adding or changing content.
I realise this is a corner case example, but it doesn't sound beyond the realms of possibility to me :)
What do you all think?

I don't know the code so don't shoot me if this isn't the case, but... If the scenario creator uses {c:} references for all of the tile, card, and monster names, shouldn't it be simple to just do those FFG name lookups on a different language to the current one?
You could have one language option for the scenario itself and another for the components.

Thatz the theory yeh :)

I think the situation may not be that uncommon. I have the same "problem": I'm a Pole but I bought the English version of the MoM game so I can play with the English language people as I live abroad.
Although I also don't know the code standing behind the app, it's doable. It should require a few things:

hash-table/array/object/dictionary (some structure) containing tiles' names for each language (as @EwenDC suggested); I presume it already exists but it may requires some changes in the structure,
changes in Select Content's design for language selection purposes,
create combo-box for language selection for each scenario,
create a machanism to display the correct tile's name.

Regarding 2. and 3., my proposition:

PSD: language_selection.zip
Personally, I think each line should have the same design and placement, therefore I spreaded and streached each line as well as aligned the text to the left and combo boxes to the right. I needed to shorten MoM's 1st Edition expansions' names.
As for the design, there are two problems related to the combo-box:

I don't know whether in Unity the combo-box is displayed within the scrollable area only or it can be displayed outside of its bounds,
the combo-box is too big; the letters and margins should be smaller but I took the list from the Options ' list.

Although I also don't know the code standing behind the app, it's doable. It should require a few things:

hash-table/array/object/dictionary (some structure) containing tiles' names for each language (as @EwenDC suggested); I presume it already exists but it may requires some changes in the structure,
changes in Select Content's design for language selection purposes,
create combo-box for language selection for each scenario,
create a machanism to display the correct tile's name.

Regarding 2. and 3., my proposition:

PSD: language_selection.zip
Personally, I think each line should have the same design and placement, therefore I spreaded and streached each line as well as aligned the text to the left and combo boxes to the right. I needed to shorten MoM's 1st Edition expansions' names.
As for the design, there are two problems related to the combo-box:

I don't know whether in Unity the combo-box is displayed within the scrollable area only or it can be displayed outside of its bounds,
the combo-box is too big; the letters and margins should be smaller but I took the list from the Options ' list.

I think the easiest thing to add this feature would be:
I would add a "Default" option, which will always simply use the currently used app language.
INI file format:
[MoMPacks]
BtT      # <- "default" language
HJ,English
SoT,Protugese
UI:
Instead of the ComboBox I would reuse this selection list from the editor:

It should automatically adjust to all screens, scroll properly etc.

The logic itself sounds fine to me.
But where would you place the button to select the default in the interface? Maybe to the right of the remove button (-)?

Default would simply be one of the options in the select list.
It's not the editor menu you showed, but the one mentioned by Chris in the Expansion Content menu section:

Ah ok makes sense. I was confused and thought the mission creator was the one who chooses the default.


feat: add climate control as climate platform

Adding climate control to the HYU/KIA integration.
This is a total Draft. My Ioniq Electric is only arriving in a few weeks and i never touched python for anything bigger than a 10-liner.
However, i would like my car to be fully controllable so i want to devote time into the HA integration. Since you guys already started the process, i would like to append to your work :)
Please let me know your thoughts and code criticisms since you also build the actual HYU/KIA Connect python lib.
As discussed in https://github.com/fuatakgun/core/pull/7, i understand basing this on top of generic_thermostat might make more sense. However before i can approach that:

i need a car, hopefully 1-2 weeks :laughing:
generic_termostat wants the toggles for heater/ac be separate switches. It also treats heater/ac as ONE toggle based on ac mode. Once i have a test bench, i can see if that mapping can be done sensibly.

@cdnninja @fuatakgun  This code now actually works - with a european Ioniq Electric. Would you test with yours cars as well? There is still a lot of rough edges though, where i have to learn the HA concepts and improve.


Some more benchmarks

This PR adds facilities to run benchmarks on larger groups.
Design

not criterion based
ability read/write group state into files for testing larger groups
different group setups
cli interface to quickly test different scenarios

This required a few small changes to OpenMLS library code to derive some more Clones for testing (benchmarking). Also, the memory key store can be written to files now.
Closes #1596

Why not use criterion? Measuring is hard and using something like criterion would at least make us avoid basic mistakes.

The memory overhead from criterion is way too high unfortunately. We'd only measure memcpy.

Since it's a benchmark, why not have it in the benches folder? I think we should reserve the examples folder for actual examples. Benchmarks can also read command line arguments, if that's your worry.

They can, but you can't run them in the same way through cargo, so you'd need to build them separately. If there's a way around that, I'm happy to move them to benches.

This needs some high-level documentation. It's hard (if not impossible) to understand what the command line arguments are for. It's also not really obvious what exactly is being measured. I'm worried it's hard for others to maintain this code.

I can add some comments. The help text on the cli should have all information for using it though.

This should run on the CI to ensure it doesn't become stale when the library is refactored. I suspect this could be run with small values so that the CI doesn't take forever but still covers all relevant code paths.

Right, I'll put on CI.

Since it's a benchmark, why not have it in the benches folder? I think we should reserve the examples folder for actual examples. Benchmarks can also read command line arguments, if that's your worry.

They can, but you can't run them in the same way through cargo, so you'd need to build them separately. If there's a way around that, I'm happy to move them to benches.

I looked into this. Passing command line arguments also works with cargo bench, but it's somewhat limited because they can conflict with libtest. You would have to resort to using env variables instead or something like that.

This needs some high-level documentation. It's hard (if not impossible) to understand what the command line arguments are for. It's also not really obvious what exactly is being measured. I'm worried it's hard for others to maintain this code.

I can add some comments. The help text on the cli should have all information for using it though.

I didn't find any explanation for what these parameters do exactly.

You would have to resort to using env variables instead or something like that.

Right, that's why I moved it over. I think it's nicer to have a proper tool.
I added usage infos and a CI job.


Added convert to UTC

@gjunge would you be so kind as to review and approve?

@mr-KVA  that should fix it! Thanks.


cyctl interactive template auth rule creation

The scope of the issue is to add the interactive mode when creating template auth rules.
If none of the flags repo, username, or password are provided, a template can be created through the interactive mode. The user will be prompted to provide details about their template auth rule one by one.
You can look into integrating promptui or some other alternative for interactive CLI mode.

@petar-cvit I would like to work on this issue and I have a doubt here, currently we are getting the username and password as values and directly validating it so I need to check beforehand that they are empty and use the values from the interactive else we need to go with the existing implications?
@quest-bot embark


Add database based login form

Description
closes #5558
Self Check:
Strike through any lines that are not applicable (~~line~~) then check the box

[ ] Attached issue to pull request
[ ] Changelog entry
[ ] Code is clear and sufficiently documented
[ ] Sufficient test cases (reproduces the bug/tests the requested feature)
[ ] Correct, in line with design
[ ] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )

https://github.com/inmanta/web-console/assets/113331659/1987f951-62b4-443b-b209-d8fbf270c972

can we show another message in case the user is unauthenticated? Like, a please wait you are being redirected? Since we know the error message before it hits the screen, I think it might be worth the effort to look further into it.

can we show another message in case the user is unauthenticated? Like, a please wait you are being redirected? Since we know the error message before it hits the screen, I think it might be worth the effort to look further into it.

It's being shown "Redirecting to log in page shortly..." (0:03)

You mixed both authentication ways into a single controler, I find the approach rather confusing. For the sake of separating concerns, it might have been more adequate to have two separate handlers that do not mingle. It makes it easier to maintain/replace/adjust.

I personally think that is fine to have them both together, as both things are responsible for authentication and if auth is used it's either one or the other.

The auth object is present on the window, you should be able to access it as soon as the app is hitting the browser. (This might mean you need to extend the typing because of Typescript) But this means you know at the root level, when you are initializing your dependencies if you are going to need a localAuth, or attempt the keycloak one.

Isn't that handled currently in AuthController?

There are very few comments in the code too, it would be nice to have more of them, at least on methods and components declaration.

Could you point parts that aren't clear? I added docstrings to the functions but I think that rest is self-explanatory, but I can be mistaken

I also don't see any E2E tests. Either make a ticket, or add them :)

Forgot about, I'll either create separate ticket or wait with merge and do it after I resolve notification bug.

Lastly, I noticed you opted for the cookies instead of the sessionStorage, can you stave your choice? Both are valid, I'd just like to know your reasoning, and then we can make up an adr for it.

as far as I know cookies are safer to store jwt that sessionStorage


Rename XEO.Camera#view to XEO.Camera#lookat

The viewing transform is always a XEO.Lookat (or subtype) because we use eye and look positions for various essential things within the scene and renderer.
Thinking about this for a bit first though, because we might be able to derive eye and look from matrices made by other potential viewing transforms.
However, it would be nice to be able to access viewing component like this:
myCamera.lookat.eye = [2,3,4];
myCamera.lookat.look = [0,0,0];

Won't fix - there may be other types of viewing transform


WIP: Use Selenium to Enable Javascript / Real-Browser Scraping + Misc Fixes

A variety of issues have recently arisen due to Twitter disabling their "Legacy" API, breaking twitterscraper:

https://github.com/taspinar/twitterscraper/issues/301
https://github.com/taspinar/twitterscraper/issues/299
https://github.com/taspinar/twitterscraper/issues/298
https://github.com/taspinar/twitterscraper/issues/296

To fix this, I re-implemented query.py using Selenium, allowing twitterscraper to programatically control a background (headless) Firefox instance.
Additionally, I refactored query.py (now query_js.py) to be a bit cleaner.
Based on my testing, this branch can successfully download tweets from user pages, and via query strings.
How to run
Please test this change so I can fix any bugs!

clone the repo, pull this branch
install selenium dependencies (geckodriver and firefox) https://selenium-python.readthedocs.io/installation.html
enter twitterscraper directory, python3 setup.py install
run your query

If you have any bugs, please paste your command and full output in this thread!
Improvements

Fix twitterscrapers failure due to twitter retiring legacy endpoints
now multiple data points are retrieved, not just tweets, this includes user metedata, location metadata, etc. All these datapoints are sent to the browser and returned by get_query_data (all tweets / metadata from a specific query) and get_user_data (all tweets / metadata on a users page).
Refactor query.py to be more clean
previously --user wouldn't get all of a users tweets and retweets due to a limitation in twitters scrollback for a given user. Now a workaround enables retrieving of tweets and retweets for a specific user via a custom search: f'filter:nativeretweets from:{from_user}'
fix https://github.com/taspinar/twitterscraper/issues/238 query_user_info broken
fix https://github.com/taspinar/twitterscraper/issues/278

Notes

pos was removed - now the browser is used to store pos state implicitly
--javascript and -j now decide whether to use query.py or query_js.py

Problems

~limit no longer works, though this should be relatively easy to fix if sufficiently desired~ (limit has now been implemented
query_user_info and query_user_page haven't been converted to use selenium, they don't work  right now. However, this data is returned as part of the metadata mentioned in Improvements bullet 2
This change requires installing selenium and geckodriver which is more difficult than just pip install. However use of docker can alleviate this.
Being that this uses a real browser, it will be slower (~1/2 as fast in my observations) and require more memory
This changes the structure of the returned json object to match twitters response. On the plus side, it allows access to much more data than before.

Oh, that's amazing! Does multiple proxies also work with Firefox? I had tested with Chrome and couldn't get it to work.

@AllanSCosta a new driver is created for each process in the pool, and each driver is initiated with a unique proxy.

Beautiful, thanks!!

@AllanSCosta users were having trouble due to twitter dropping their legacy endpoints, see the linked issues.

Sorry. I do not fully understand.  Does this work? Thanks.

A variety of issues have recently arisen due to Twitter disabling their "Legacy" API, breaking twitterscraper:

#301
#299
#298
#296

To fix this, I re-implemented query.py using Selenium, allowing twitterscraper to programatically control a background (headless) Firefox instance.
Additionally, I refactored query.py (now query_js.py) to be a bit cleaner.
Based on my testing, this branch can successfully download tweets from user pages, and via query strings.
How to run
Please test this change so I can fix any bugs!

clone the repo, pull this branch
install selenium dependencies (geckodriver and firefox) https://selenium-python.readthedocs.io/installation.html
enter twitterscraper directory, python3 setup.py install
run your query

If you have any bugs, please paste your command and full output in this thread!
Improvements

Fix twitterscrapers failure due to twitter retiring legacy endpoints
now multiple data points are retrieved, not just tweets, this includes user metedata, location metadata, etc. All these datapoints are sent to the browser and returned by get_query_data (all tweets / metadata from a specific query) and get_user_data (all tweets / metadata on a users page).
Refactor query.py to be more clean

Notes

pos was removed - now the browser is used to store pos state implicitly

Problems

limit no longer works, though this should be relatively easy to fix if sufficiently desired
query_user_info and query_user_page haven't been converted to use selenium, they don't work  right now. However, this data is returned as part of the metadata mentioned in Improvements bullet 2
This change requires installing selenium and geckodriver which is more difficult than just pip install. However use of docker can alleviate this.
Being that this uses a real browser, it will be slower (~1/2 as fast in my observations) and require more memory

Thank you. I wanted to try the project before changing the files and it worked (very interesting). Then I tried the codes you edited. It working smoothly.

@AllanSCosta users were having trouble due to twitter dropping their legacy endpoints, see the linked issues.
Thank you,

I get an error like this:
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
Which file do i need to edit?

I got error like this:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities

I got error like this:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities

Problem solved. I forgot to get Firefox installed...😂

If you don't mind me asking,

@AllanSCosta users were having trouble due to twitter dropping their legacy endpoints, see the linked issues.
Thank you,

I get an error like this:
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
Which file do i need to edit?

You need to install Geckodriver. If it's a mac, brew install geckodriver should suffice.

@AllanSCosta retries < 0 will execute if the conditions which decrement retries within the loop are met.

 I just pushed those changes, reverted!

Fun side note: if you want to see the browsers in actions (or if theres an issue see what's going wrong) allow the browser to be visible by setting driver.headless = False here https://github.com/taspinar/twitterscraper/pull/302/files#diff-83a91a4e1920f0a97f5f9b7c5eabefc5R48

Hi @lapp0, I'm still debugging some stuff here. For some reason, the response is proper (200) and I do manage to get data, but in query_single_page the array relevant_requests ends up always empty.

@AllanSCosta I could not reproduce.

As an aside, it appears that scrolling down on twitter stops after 1300 tweets on realDonaldTrumps page. I'll investigate how to continue scrolling.

https://github.com/taspinar/twitterscraper/pull/304/files appears to fix the main issue. I am going to make js optional here so we can have a backup if/when #304's solution fails.

I ran the code tweets = get_user_data('realDonaldTrump') and got 0 tweets.

@AllanSCosta @pumpkinw can you please

add driver.get_screenshot("foo.png") to this line https://github.com/taspinar/twitterscraper/pull/302/files#diff-83a91a4e1920f0a97f5f9b7c5eabefc5R126

then share the resulting screenshot

share your geckodriver version
share your firefox version
share your operating system and version
share your selenium version

@lapp0
The screenshot correctly depicts Trump's twitter (as if I had manually opened the browser and accessed it). Here are the versions:
geckodriver 0.26.0
Firefox 77.0.1 (64-bit)
OS and version macOS Mojave 10.14.5
Selenium 3.141.0

thanks @AllanSCosta
Are you using selenium-wire==1.1.2? It appears I'm using a dated version (0.7.0), as I was able to reproduce this problem by upgrading to 1.1.2.

I'm using seleniumwire version 1.1.2 indeed :)

Please try now, I have pegged selenium-wire to 1.0.1

It works now, thanks!!

Please try now, I have pegged selenium-wire to 1.0.1

Thanks! It works for me now!

@AllanSCosta  Yes, version 1.0.2 of selenium doesn't do conversion from gzip bytes -> python object.

Thank you so much, but I'm kinda lost? I'm new to this and I can't seem to pull your branch from my github desktop. I've installed gecko and selenium, but I didn't understand exactly what I have to do to run the query with your changes. Sorry if it's too much trouble!

thanks for testing @barabelha ! To run with my changes, you must add the --javascript argument.

@bamboozooled #304 is in origin/master which fixes the legacy issue for now. It isn't on pip though.
This PR isn't in master either, it's still open.
@taspinar what are the procedures to get a new pip version? Do we just need a new verison tag on git? I think #304 is an important change to get to pypi since it fixes the program.

Thanks a lot @lapp0 !

Hi @lapp0 , I am new at using github so I wanted to know if you could give me more details of how to run  "clone the repo, pull this branch" because I'm getting the same problems of getting 0 tweets when using the twitterscraper. Thank you!!

@Michelpayan

Install git
run the commands

git clone https://github.com/lapp0/twitterscraper.git
git checkout origin/selenium

change directories twitterscraper, then run python3 setup.py install along with the other install instructions in the post.
Let me know if you have any questions.

Hi @lapp0
I set this up just as instructed, but I keep getting this error:
Traceback (most recent call last):   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\Scripts\twitterscraper-script.py", line 33, in <module>     sys.exit(load_entry_point('twitterscraper==1.4.0', 'console_scripts', 'twitterscraper')())   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\twitterscraper-1.4.0-py3.7.egg\twitterscraper\main.py", line 118, in main   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\twitterscraper-1.4.0-py3.7.egg\twitterscraper\query_js.py", line 165, in get_query_data   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\twitterscraper-1.4.0-py3.7.egg\twitterscraper\query_js.py", line 186, in retrieve_data_from_urls   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\billiard-<IP_ADDRESS>-py3.7.egg\billiard\pool.py", line 1969, in next     raise Exception(value) Exception: Traceback (most recent call last):   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\billiard-<IP_ADDRESS>-py3.7.egg\billiard\pool.py", line 362, in workloop     result = (True, prepare_result(fun(*args, **kwargs)))   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\twitterscraper-1.4.0-py3.7.egg\twitterscraper\query_js.py", line 79, in query_single_page     driver = get_driver(proxy)   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\twitterscraper-1.4.0-py3.7.egg\twitterscraper\query_js.py", line 40, in get_driver     profile = webdriver.FirefoxProfile()   File "C:\Users\Administrator\Documents\twitterscraper\js\venv\lib\site-packages\selenium-4.0.0a6.post2-py3.7.egg\selenium\webdriver\firefox\firefox_profile.py", line 59, in __init__     WEBDRIVER_PREFERENCES)) as default_prefs: FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Administrator\\Documents\\twitterscraper\\js\\venv\\lib\\site-packages\\selenium-4.0.0a6.post2-py3.7.egg\\selenium\\webdriver\\firefox\\webdriver_prefs.json'
I have installed both firefox and geckodriver.
I used this on the command-line: twitterscraper realDonaldTrump -j -c -o output.csv -ow
And this in Python IDE: js = get_user_data('realDonaldTrump')
Same issue.
Please let me know what I am doing wrongly.

@smuotoe Not sure, it seems some users have had problems in the past with this https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/6808
However you are on latest unstable. Perhaps the same mistake was made there? Could you try version 3.141.59?

i got this error:
Can't load /home/ml/.rnd into RNG<PHONE_NUMBER>72320:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:98:Filename=/home/ml/.rnd
and after couple of min.:
Exception happened during processing of request from ('<IP_ADDRESS>', 50790) Traceback (most recent call last):   File "/usr/lib/python3.6/socketserver.py", line 654, in process_request_thread     self.finish_request(request, client_address)   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request     self.RequestHandlerClass(request, client_address, self)   File "/usr/local/lib/python3.6/dist-packages/selenium_wire-1.0.1-py3.6.egg/seleniumwire/proxy/proxy2.py", line 65, in __init__     super().__init__(*args, **kwargs)   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__     self.handle()   File "/usr/lib/python3.6/http/server.py", line 420, in handle     self.handle_one_request()   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request     method()   File "/usr/local/lib/python3.6/dist-packages/selenium_wire-1.0.1-py3.6.egg/seleniumwire/proxy/handler.py", line 127, in do_GET     super().do_GET()   File "/usr/local/lib/python3.6/dist-packages/selenium_wire-1.0.1-py3.6.egg/seleniumwire/proxy/proxy2.py", line 224, in do_GET     self.wfile.write(res_body)   File "/usr/lib/python3.6/socket.py", line 604, in write     return self._sock.send(b)   File "/usr/lib/python3.6/ssl.py", line 944, in send     return self._sslobj.write(data)   File "/usr/lib/python3.6/ssl.py", line 642, in write     return self._sslobj.write(data) ConnectionResetError: [Errno 104] Connection reset by peer

@smuotoe Not sure, it seems some users have had problems in the past with this SeleniumHQ/selenium-google-code-issue-archive#6808
However you are on latest unstable. Perhaps the same mistake was made there? Could you try version 3.141.59?

Thanks, it seems the latest version has a bug. It works with a lower version.

Hi @lapp0
Thanks for this update to twitterscraper and your help in other areas. MR #304 has already been merged and is available in version 1.5.0. twitterscraper is now at version 1.6.0 which also includes other updates.
Let me test his branch tonight and see if we can make it available as version 2.0.0

Hi @lapp0
I tried to follow the procedures described on this page as much as possible. I implemented the fix for dateranges in a local version.
Whenever I use the -jarguemnt, I get consistently a lower number of tweets than the indicated limit (zero number of tweets or a few tens number of tweets).
I have installed Firefox 78.0.2 on Mac,
GeckoDriver 0.26.0
Selenium version 3.141.0,
MacOS Catalina 10.15.5

@taspinar I have a test case that I'll push to this branch. Still working on figuring out why this version has fewer tweets.

Remaining work:

[ ] Convert dict to Tweet
[ ] Update Dockerfile to install geckodriver and firefox
[ ] fix test case issue where fewer tweets are returned that expected

Hi @lapp0, has there been any progress on this so far? Does your version still work after the recent changes on the Twitter website (see #336, #339, #343, #344, #337, twintproject/twint#604, Mottl/GetOldTweets3#98)? If yes, your work would be very valuable for us from @Museum-Barberini-gGmbH.

I can confirm that this method works quite well enough. I am wondering if using helium won't solve the geckodriver installation issue? As helium ships with common webdrivers.
See helium https://github.com/mherrmann/selenium-python-helium

@LinqLover this branch works in that it retrieves tweets using a headless browser successfully, emulating a real user scrolling. However, it doesn't convert the results to a Tweet object. I've been quite busy lately and haven't had a chance to address this (additionally, another user came up with a fix making this branch less urgent for the time being for a while).
Additionally, the API returns tweets that aren't in the given daterange. When you search for a tweet in a given daterange, you might see tweets that reply to a tweet outside of the daterange, and this endpoint includes those tweets in the set.
So to get this branch merged, we need to fix the extra tweets and convert dict to Tweet fixes in.

@taspinar what are these #hack? items supposed to be?
        tweets.append(Tweet(
            screen_name=user['screen_name'],
            username=user['name'],
            user_id=tweet_item['user_id'],
            tweet_id=tid,
            tweet_url=f'https://twitter.com/{user["screen_name"]}/status/{tid}',  # hack?
            timestamp=timestamp_of_tweet(tweet_item),  # hack?
            timestamp_epochs=timestamp_of_tweet(tweet_item),  # hack?
            text=tweet_item['full_text'],
            text_html=None,  # hack?
            links=tweet_item['entities']['urls'],
            hashtags=tweet_item['entities']['hashtags'],
            has_media=None,  # hack?
            img_urls=None,  # hack?
            parent_tweet_id=tweet_item['in_reply_to_status_id'],
            reply_to_users=tweet_item['in_reply_to_user_id'],  # hack?
        ))

This is what data['tweets'].items()]0] looks like:
('932325048706387969', {'created_at': 'Sun Nov 19 19:09:48 +0000 2017', 'id':<PHONE_NUMBER>06387969, 'id_str': '932325048706387969', 'full_text': 'Sonic Burger with alphabet soup for dipping', 'truncated': False, 'display_text_range': [0, 43], 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': []}, 'source': 'Secret Menu Item Generator', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user_id':<PHONE_NUMBER>03927552, 'user_id_str': '775339094603927552', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 0, 'favorite_count': 0, 'reply_count': 0, 'quote_count': 0, 'conversation_id':<PHONE_NUMBER>06387969, 'conversation_id_str': '932325048706387969', 'favorited': False, 'retweeted': False, 'lang': 'en', 'supplemental_language': None})

I fixed the daterange issue and Tweet issue (partially) and pushed.

@JiGGie145 I'm not familiar with helium, but it looks quite useful given the issues users in this thread have been having. Would you like to submit your own pull request targeting this one that includes helium?

Hi @lapp0, your work is beautifull, but not work for me :c
Mozilla Firefox 80.0.1 on MAC
geckodriverVersion': '0.27.0'
selenium:                   3.141.0
selenium-wire:              1.0.1
macOS 10.15.6
my code is
import twitterscraper
tweets = twitterscraper.get_user_data('realDonaldTrump',poolsize=1)
print(tweets)
and when open the explorer Firefox, the message is the next

please help. Will it have to do with the proxy?

Thanks @christiangfv
On the surface that appears to be an issue with one or many proxies.

have you tried running it a second time?
Have you tried running twitterscraper with proxies disabled?

Thanks for your attention @lapp0

Yes, I have tried several times, even restarting the machine.
how i can disable the proxies?

In my terminal
INFO: Using proxy <IP_ADDRESS>:31158
INFO: Scraping tweets from https://twitter.com/search?f=live&vertical=default&q=from:realDonaldTrump since:2020-09-19 until:2020-09-21&l=
INFO: Using proxy <IP_ADDRESS>:80
INFO: Got 0 data (0 new).
INFO: Scraping tweets from https://twitter.com/search?f=live&vertical=default&q=filter:nativeretweets from:realDonaldTrump since:2020-09-18 until:2020-09-19&l=
INFO: Using proxy <IP_ADDRESS>:8080
INFO: Got 0 data (0 new).

@christiangfv try --disableproxy

You may be experiencing this issue https://github.com/wkeeling/selenium-wire/issues/55#issuecomment-511182605

not working --disableprxy nor -dp. I try in terminal commands, for example: twitterscraper Trump --disableproxy --limit 10 --output=tweets.json.
And when I modify the get_driver() function, the same thing happens

@christiangfv please try what I just pushed

@lapp0 it doesn't work for me, and now it also doesn't open explorer when I switch opt.headless = False.
Am I doing something wrong?

I can confirm that this method works quite well. I am wondering if using helium won't solve the geckodriver installation issue? As helium ships with common webdrivers.
See helium https://github.com/mherrmann/selenium-python-helium

I tried this suggestion from a friend and was able to make the geckodriver work (in Ubuntu 18.04 with Firefox 80.0.1):
wget https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-linux64.tar.gz
tar -xvf geckodriver-v0.27.0-linux64.tar.gz
sudo mv geckodriver /usr/bin

@lapp0 does not change, I have the same 502 error. I try it with proxy, no proxy, in firefox and chrome driver and it's the same

@christiangfv thats unfortunate. Perhaps someone more talented than me could update the docker image to install selenium so we don't have to deal with these platform-specific issues.

Yes @lapp0 , I have done it, now I will test in a virtual machine with Ubuntu.

@lapp0 Works in linux Ubuntu!! Ubuntu is always a faithful ally. But I have a new problem, you know some this?
mi code:
tweets = twitterscraper.get_query_data('realDonaldTrump',poolsize=1,begindate=dt.date(2020, 9, 20),limit=10)
print(tweets)
and this function return a void list. but when I debug, I can check that take a correctly tweet.

Thanks for testing @christiangfv !
I made a mistake in the date filtering function I pushed today so it doesn't allow single-day ranges. Pushing a fix.

With this you can solve

Thanks @lapp0, Hey friend, do you know if this implementation works on a server with ubuntuServer or other?

@christiangfv Good to hear!
I don't see any reason why this wouldn't work on ubuntu server

@AllanSCosta a new driver is created for each process in the pool, and each driver is initiated with a unique proxy.
This uses FirefoxDriver, but I think ChromeDriver would work for this too.

I got this problem and i don't know why
I use "From within Python"

I got this problem and i don't know why
I use "From within Python"

I had run this, but it seemed no effect

@Buaasinong Are you in the correct directory? What is your operating system? in linux you need the sudo command.

Are you in the correct directory? What is your operating system? in linux you need the sudo command.

Oh, i'm using windows.And i think my directory is correct.

Are you in the correct directory? What is your operating system? in linux you need the sudo command.

Oh, i'm using windows.And i think my directory is correct.

I try to run setup.py directly ,and get this

@Buaasinong The correct command for the installation is Python3 setup.py install. You need add argument install

@Buaasinong The correct command for the installation is Python3 setup.py install. You need add argument install

Yes! I used python setup.py install(because python3 setup.py install is still no result).But it still don't work. Could you give me a picture about successfully running in terminal?

@Buaasinong The correct command for the installation is Python3 setup.py install. You need add argument install

Yes! I used python setup.py install(because python3 setup.py install is still no result).But it still don't work. Could you give me a picture about successfully running in terminal?

OH，I think python setup.py install is not useful.sad

@Buaasinong are you running python2? What is python --version?
Would be really helpful if someone got the docker working with selenium so we could avoid these version and platform specific issues.

Hi @lapp0, I have noticed that when I retrieve the tweets, they do not appear in order. I only need the last 20 tweets from an account in order. For this I implement changes in

and the new link that i used is:
INIT_URL = 'https://twitter.com/search?q={q}&src=typed_query&f=live'
and finally, for sorted tweets, I sorted previusly the dict  with tweets in:

@Buaasinong are you running python2? What is python --version?
Would be really helpful if someone got the docker working with selenium so we could avoid these version and platform specific issues.

python 3.7 ，but it‘s 32bits

Hi Bro ,I got a new problem like you in browser @christiangfv ,and I want to disable proxy for using --disableproxy,there is an error in my terminal

Hi @Buaasinong, the problem is that the --disableproxy or -dp argument is not implemented. I had the problem on macOS 10.15.6, I think it could be system restrictions. I solved it by installing on Ubuntu linux

--disableproxy hasn't been merged into this branch yet, but is in master. However, I don't think that's the problem. I think it's a problem specific to selenium that don't involve the proxies at all.
I have pushed untested docker changes to install geckodriver, however they're pretty straightforward. @Buaasinong could you try to use the Dockerfile to build and run?

Bug find:
The following query has variable results: https://twitter.com/search?q=foo+bar+since%3A2017-11-11+until%3A2017-11-13
Sometimes it includes foobar is being so I'm out the first round and sometimes it doesn't.
I can't figure out what conditions result in this being the case, sometimes there are 20 results, sometimes there are 40, sometimes 57. Perhaps the only option is to require multiple passes, unfortunately.

--disableproxy hasn't been merged into this branch yet, but is in master. However, I don't think that's the problem. I think it's a problem specific to selenium that don't involve the proxies at all.
I have pushed untested docker changes to install geckodriver, however they're pretty straightforward. @Buaasinong could you try to use the Dockerfile to build and run?

Thanks Bro, I will try

Hi @Buaasinong, the problem is that the --disableproxy or -dp argument is not implemented. I had the problem on macOS 10.15.6, I think it could be system restrictions. I solved it by installing on Ubuntu linux

Wow, It's a bad news.But still appreciate.

Upon running a query, I get the following error: "ModuleNotFoundError: No module named 'twitterscraper.ts_logger"

@Bakdaulet1 I pushed a broken change on accident. Most recent push should resolve.

I ran this 100 times:
query_js.get_query_data(begindate=dt.date(2019, 11, 11), enddate=dt.date(2019, 11, 13), poolsize=3, lang='en', query='foo bar')
I then counted the number of tweets recieved each time. Here is the mapping of "number of tweets" -> "number of times this number of tweets were returned"
41: 9
55: 16
70: 20
73: 1
74: 1
83: 3
84: 50

So exactly half the time we got 84 tweets.
I'm running this experiment again with multiple passes for each day (unfortunate but probably workable solution), will update this comment with results.

https://github.com/lapp0/twitterscraper/blob/faf75afb7eff15a04df471a0b651011b67db85fa/Dockerfile#L4 so the firefox install here in the docker file doesnt work  as is , https://hub.docker.com/r/selenium/node-firefox/dockerfile should perhaps maybe use the officially maintained selenium//firefox dockerfile as a base ??

@webcoderz you probably know docker better than me. Would you be willing to make the change?

would you recommend doing 2 docker services communicating over a docker network vs one container vs single container?

@webcoderz IMHO it'd be better to have a single docker container.

I am getting the following error:
INFO:twitterscraper:Scraping tweets from https://twitter.com/Bakdaul52729738
INFO:twitterscraper:Using proxy <IP_ADDRESS>:8080
INFO:twitterscraper:Got 0 tweets from username Bakdaul52729738

Number of passes -> % of time it collects every single tweet
get_query_data(begindate=dt.date(2019, 11, 11), enddate=dt.date(2019, 11, 13), poolsize=3, lang='en', query='foo bar')

4 passes: 100%
3 passes: 80%
2 passes: 65%

get_query_data(begindate=dt.date(2018, 5, 5), enddate=dt.date(2018, 5, 7), poolsize=3, lang='en', query='foo bar')
(running, will update)

@Bakdaulet1 please share your command/code

twitterscraper Bakdaul52729738--user -o tweets_username.json

@Bakdaulet1 please share your command/code

twitterscraper Bakdaul52729738--user -o tweets_username.json

@Bakdaulet1 the legacy endpoint is broken. To use the changes in this merge request you need the --javascript (or -j) argument.

query_js.get_user_data(from_user="webcoderz")
^CTraceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 254, in retrieve_data_from_urls
for new_data in pool.imap_unordered(partial(query_single_page, limit=limit_per_pool, use_proxy=use_proxy), urls):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1969, in next
raise Exception(value)
Exception: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 362, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 95, in query_single_page
driver = get_driver(proxy)
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 54, in get_driver
seleniumwire_options={'verify_ssl': False}
File "/usr/local/lib/python3.7/site-packages/seleniumwire/webdriver/browser.py", line 45, in init
super().init(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in init
keep_alive=True)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: invalid argument: can't kill an exited process

do you have a few examples of how to do it programmatically , i get that when running the get user info and the other query you are testing with.

would also be nice to have a function that scrapes by twitter id....
you can use this
twitter_id="1308322643054034944"
"https://twitter.com/anyuser/status/{}".format(twitter_id)
https://twitter.com/anyuser/status/1308322643054034944
will take you to the correct tweet , alls you need is the twitter id

@webcoderz good idea! Although I think that's out of scope for this merge request. Feel free to make your own.

ok had found a error in my Dockerfile will try again and let you know, just pushed the change and PR (had symlink to just geckdriver but not to selenium wire

File "/app/test.py", line 10, in test_simple_js
res = query_js.get_query_data(**call_dict)
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 180, in get_query_data
data = retrieve_data_from_urls(urls, limit=limit, poolsize=poolsize, use_proxy=use_proxy)
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 257, in retrieve_data_from_urls
for new_data in pool.imap_unordered(partial(query_single_page, limit=limit_per_pool, use_proxy=use_proxy), urls):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1969, in next
raise Exception(value)
Exception: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 362, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 95, in query_single_page
driver = get_driver(proxy)
File "/usr/local/lib/python3.7/site-packages/twitterscraper/query_js.py", line 54, in get_driver
seleniumwire_options={'verify_ssl': False}
File "/usr/local/lib/python3.7/site-packages/seleniumwire/webdriver/browser.py", line 45, in init
super().init(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in init
keep_alive=True)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: invalid argument: can't kill an exited process ok yea same error.

@lapp0 submitted PR that dockerfile tested works,  puts executable in /bin for geckodriver, was wondering can also add the other browser drivers as well if u wanted to do rotating browsers for improved proxying capability or something like that

IMO, adding new browsers will just increase complexity with little to no benefit. I don't think twitter is going to throttle/ban Firefox.
The biggest problem right now is memory and CPU, so if chrome is more efficient browsing twitter, we might want to go that way. Although, I think installing ublock origin in firefox would be a better decision for that.

Ran into this for a big query:

Going to experiment with a throttling mechanism avoid

i think can do that all via docker too will look into it..

That'd be great, we might even be able to set custom ublock rules to block irrelevant twitter endpoints and speed up scraping.

@lapp0 I noticed this in the latest commit... And I am curious, with the return keywords in the if...else conditional statement, will the code ever go further to retry?

@smuotoe Yeah, that entire function needs to be cleaned up. It's slowly built technical debt as I've handled more and more edge cases.
Good catch, pushing a fix.

https://blog.mozilla.org/addons/2019/10/31/firefox-to-discontinue-sideloaded-extensions/

@webcoderz thats too bad. Do you know if chromium has these capabilities?

https://stackoverflow.com/questions/34222412/load-chrome-extension-using-selenium something like that should work   using selenium natively to do it

@webcoderz thanks for researching!

I just pushed a commit with a lot of improvements

proxies now work with selenium-wire
we find the sqrt(N) fastest proxies and use them. Scraping is observably faster.
code is refactored, cleaned up, and has better error handling / retrying parameters. Is less prone to failure.

Writing a few more test cases and handling some edge cases, then I think this will be ready to merge.

I just pushed a commit with a lot of improvements

proxies now work with selenium-wire
we find the sqrt(N) fastest proxies and use them. Scraping is observably faster.
code is refactored, cleaned up, and has better error handling / retrying parameters. Is less prone to failure.

Remaining work:

[ ]  the main problem: rate limiting by twitter. Perhaps we need a more extensive proxy list? Perhaps different browser profiles might help?
[ ]  write more test cases
[ ]  use a larger proxy list
[ ]  cache the proxy priority list so we aren't querying 100s of proxies each run
[ ]  handle misc edge cases

You are so great!
Hope you will fix the problem and make twitterscraper run more smoothly soon.

Thanks! Your work on docker has been great too!

Exception Message: Timeout loading page after 10000ms  while requesting "https://twitter.com/search?f=live&vertical=default&q=realDonaldTrump since:2020-01-31 until:2020-02-01" Traceback (most recent call last):   File "C:\Users\tempSomto\OneDrive\twitterscraper-lappo\query_js.py", line 64, in retrieve_twitter_response_data     driver.get(url)   File "C:\Users\tempSomto\OneDrive\twitterscraper-lappo\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get     self.execute(Command.GET, {'url': url})   File "C:\Users\tempSomto\OneDrive\twitterscraper-lappo\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute     self.error_handler.check_response(response)   File "C:\Users\tempSomto\OneDrive\twitterscraper-lappo\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response     raise exception_class(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 10000ms
Hi @lapp0 great work on this thus far. I tested the latest commit and it seems there is something wrong (see above error log).
Also, I noticed the TimeoutException that was imported in query_js was not used in the file.

@smuotoe ya, I set a strict timeout because some proxies are pretty slow, so we want to restart with a new proxy if it takes too long loading twitter. I changed it to 30 seconds though.

whats the current status of the examples @lapp0 ? i havent been able to get anything back with the example in the unit test

Hi, any updates on this PR? Can it already be used for production or is the rate limits problem too massive?
I just did some research on proxy list and I think this package sounds quite promising: https://pypi.org/project/proxyscrape/
Usage:
import proxyscrape

def get_proxies():
    collector = proxyscrape.create_collector('default', 'http')
    return collector.get_proxies()

Could this help?

Hi, I am new to this. I tried to follow comments in this branch, but I still got 0 tweets.
Code:
`
from twitterscraper import query_js
import datetime as dt
import pandas as pd
if name == 'main':
tweets = query_js.get_query_data(begindate=dt.date(2019, 11, 11), enddate=dt.date(2019, 11, 12), poolsize=5, lang='en',
query='foo bar')
df = pd.DataFrame(t.dict for t in tweets)
print(tweets)
print(df)
`
console.txt
In the console, there are three errors occurred during the run:
1.TypeError: the JSON object must be str, bytes or bytearray, not list
2.selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 30000ms
3.ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
What can I do to this?

I'm having the same problem as @edmangog:
>>> import twitterscraper as ts
>>> ts.query_tweets('obama', limit=10)
[]

If I enable logging, I see lots of INFOs about "Got 0 tweets" but no warnings or errors.
Anyone having an idea why? Did they just ban Selenium bots, too? :-(

Sorry for the delayed response, I've been quite busy with professional work lately.
Errors
@edmangog Your error is likely due to twitter throttling.

Core Rate Limiting Problem
Some recent experimentation has indicated two things:

We can shard queries by including / excluding common words (e.g. foo bar -> foo bar "and" + foo bar -"and"), which can be applied to an arbitrarily large number of splits. This can reduce the error rate for high volume queries significantly.  For example, I previously could not download all mentions of "Donald Trump" for a single day, however this new technique allows shards to be retrieved successfully.

Twitter blocks high volume requests from a single IP.

Since we're all using the same proxy list, our bandwidth is collectively limited. I can retrieve significantly more tweets without being throttled on my local IP than with the shared proxy list, however, they are throttled regardless.
Further experimentation may find ways to stretch the usability of these proxies. I'm not  even sure what the exact rate-limits are per IP, and knowing that will be valuable. Regardless, a single proxy will always hit its limits.
Solution and Implementation
I think the only solution here is to use "personal" proxy servers. This would practically make paid cloud services a requirement for twitterscraper, which may be a necessary evil.
As I mentioned I am quite busy with my professional work, but I will dedicate some time this late November stabilizing this branch and making it compatible with a "custom proxy list". Additionally I will need to write instructions for ad-hoc proxy generation.
Thanks for your patience.

try a  tor proxy we fixed twints tor proxy and the latency isnt too bad! https://github.com/twintproject/twint/issues/913 heres everything for reference if you choose to go that way,

thanks for the reference @webcoderz
Could you clarify the difference between these two projects? Is there some feature in twitterscraper not present in twint?

its set up a little differently, but it seems twitter can somewhat detect twint as the last couple of ui changes completely broke it , whereas complete browser scraping cant really be detectable because if done correctly its indiscernable between scraping and actual traffic. (atleast what i think anyways..)

Hello, sorry I haven't updated this in a while.
I've tried to make this work, but unfortunately the only workable solution I've found is with a large number of unused proxies. If someone knows of a way to generate a large number of proxies cheaply, perhaps some kind of proxy as a service, please let me know.

How about twitterscraper grab data only.
Users buy proxies themselves.
Is that way more reliable than free proxies?
------------------ 原始邮件 ------------------
发件人:                                                                                "taspinar/twitterscraper"<EMAIL_ADDRESS>发送时间: 2021年2月9日(星期二) 晚上6:11
收件人<EMAIL_ADDRESS>抄送: "张志成"<2543161449<EMAIL_ADDRESS>主题: Re: [taspinar/twitterscraper] WIP: Use Selenium to Enable Javascript / Real-Browser Scraping + Misc Fixes (#302)
Hello, sorry I haven't updated this in a while.
I've tried to make this work, but unfortunately the only workable solution I've found is with a large number of unused proxies. If someone knows of a way to generate a large number of proxies cheaply, perhaps some kind of proxy as a service, please let me know.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

Right, twitterscraper should absolutely be agnostic to the source of the proxies. It's just that I'm not aware of a service I could use to test it out.

try it with tor not sure the latency aspect but tor is surefire to work
On Wed, Feb 10, 2021 at 12:37 PM lapp0<EMAIL_ADDRESS>wrote:

Right, twitterscraper should absolutely be agnostic to the source of the
proxies. It's just that I'm not aware of a service I could use to test it
out.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/taspinar/twitterscraper/pull/302#issuecomment-776923237,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEXWRAONVNW454QT5B37HFTS6LG6BANCNFSM4NTF7DSA
.

Unfortunately, with real browser scraping tor is extremely slow (30 seconds to load a new chunk of tweets vs 2 seconds for a proxy). This may be alleviated by blocking certain twitter requests though.
Either way (proxy or tor), a firefox profile which blocks unnecessary twitter requests would be helpful.


build issues on various compiler versions

on my fork I added all the compiler versions since 2.0.80 to see how the travis-ci build handles it:
https://travis-ci.org/SingingBush/mysql-native/builds/579444314
Things are not great. I've said before (#141 drop support for old compiler versions) that it's an unnecessary maintenance burden. Do you really need to support all the way back to dmd 2.068.2 (Sep 2015) to the latest dmd 2.087.1 version?

On September 1, 2019 10:44:07 AM EDT, Samael<EMAIL_ADDRESS>wrote:

on my fork I added all the compiler versions since
2.0.80
to see how the travis-ci build handles it:
https://travis-ci.org/SingingBush/mysql-native/builds/579444314
Things are not great. I've said before (#141 drop support for old
compiler versions) that it's an unnecessary maintenance burden. Do you
really need to support all the way back to dmd 2.068.2 (Sep 2015) to
the latest dmd 2.087.1 version?

Just took a quick look at several of those failures, including the one using the latest dmd. They all still compiled (just with one non-fatal deprecation that looks trivial to fix without affecting backwards compatibility). The failures I do see are failures to connect to the socket, which in my experience pretty much always indicates some sort of configuration problem, like an incorrect connection string or db server not running for some reason. It's highly unlikely for compiler/phobos changes to cause this.

I've done a little digging. Seems there is an issue with travis-ci and newer versions of Ubuntu/Mariadb that I hope to have sorted now. Will have to see how this build does:
https://travis-ci.org/SingingBush/mysql-native/builds/579489019

Now fixed in master.


Fixes moment.js library load order.

@DanielJMaher This is a critical fix to the mission load page.  Fix applied to erd21.


Update citation in README to ICCV version.

I image you would like people to cite the ICCV version of the paper rather than the preprint :)

Cool! Thanks


feat: add SUPER permission for setData and execute

What does this PR introduce?

NB awaiting for #134 to be merged first

remove bypass permissions check if caller has ALLPERMISSIONS, introduced in #123
As this introduced a bug patched in #128
Replace the bypass ALLPERMISSION feature by the following:
Introduce new set of permissions in the KeyManager: SUPER_SETDATA, SUPER_CALL, SUPER_DELEGATECALL and SUPERSTATICCALL to allow not checking the list of addresses / functions / standards and ERC725YKeys.

Closed and reopened in #150 to have a cleaner git history.


Wrong search rank for query language reference

The first search result on bing for "application insights query language reference" is
https://docs.microsoft.com/en-us/azure/application-insights/app-insights-analytics
when it should be
https://docs.loganalytics.io/docs/Language-Reference

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 41dcdb9d-ab1f-5703-8d0f-20e14615a564
Version Independent ID: 119e189e-b035-e0d0-d093-c576af68917a
Content: Analytics - the powerful search and query tool of Azure Application Insights
Content Source: articles/application-insights/app-insights-analytics.md
Service: application-insights
GitHub Login: @mrbullwinkle
Microsoft Alias: mbullwin

@ostreifel thanks for bringing this to our attention. The reference content on http://docs.loganalytics.io is actually currently in the process of being migrated to its new official home on docs.microsoft.com.
While we do take into account SEO when authoring docs, we don't have exact control over how search engines (Bing included) ultimately rank our content. With that said, we do closely monitor how our content is performing and adjust and improve accordingly. The migration of the reference content is part of an effort to improve the user experience so that docs.microsoft.com is the authoritative source for information rather than having a disjointed experience with multiple sites for reference content.

@ostreifel Thank you very much for bringing this to our attention.  Echoing what @mrbullwinkle has said, we have limited control over how our content is ranked on different search engines however comments like yours will help guide others to the right place in the future.
Thank you again for your feedback!  We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


Add type restriction to private Process constructor

The unrestricted ctor argument causes the compiler to match a call like Process.new(nil) to this private ctor.
Restricting the argument to @pid : Int32 gives a more helpful compiler error, explaining that no overload matches.
class Foo
  getter value : Int32

  # private def initialize(@value)
  private def initialize(@value : Int32)
  end

  def initialize
    @value = 0
  end
end

Foo.new(nil)

# Error in line 13: no overload matches 'Foo.new' with type Nil
# Overloads are:
#  - Foo.new(value : Int32)
#  - Foo.new()

https://carc.in/#/r/5gg1
Fixes #7038

BTW, the error suggesting that Foo.new(value : Int32) is a valid ctor is kinda wrong as it is private, and can't be used from the outside, no?

Libc::PidT sounds good to me, given there is def self.pid : Libc::PidT.
I just followed the existing getter :)
I will change them both if we like.
@Sija It is used legally in the public method def self.fork, there is a call to new pid

Yeah updating both sounds good to me

I'd rather not make the Process#pid : LibC::SizeT change, I don't want to expose platform-specific integer types.

We already expose this one though, even expect it in some public API. Would you feel better about an alias at Process::PidT and using that for all public API?

I'd feel better changing all the places where we expose this to not expose this, instead of going in the other direction.

Note: I'm not suggesting we do that in this PR, we can just remove that part of the diff.

We can discuss not exposing platform-specific types in a separate issue. But the type of Process#pid is LibC::PidT, whether we declare it as an explicit return type or not. I think it's better to be explicit because it improves API documentation.


Cleanup story in RelationalTestStore

What's the cleanup story in RelationalTestStore? The database creation generally happens in Initialize method, but is there some place to do safe database drop? I personally don't think tests should leave stuff behind.

I personally don't think tests should leave stuff behind.

We leave behind databases. This may be SqlServer specific but Database creation is time-expensive step. Especially, SqlServer takes few seconds before database is ready to be used after creation. Hence we don't drop databases in code. Whenever there is issue, you can manually delete all databases and tests will recreate them. What we do instead is clean all objects from database (a fully clean database) and then recreate all objects and seed data as needed. The processing happens here.
https://github.com/aspnet/EntityFrameworkCore/blob/9be0341cfc60a852b7c30da444b460702e0a055d/test/EFCore.Specification.Tests/TestUtilities/TestStore.cs#L53-L60
Providers can override appropriate methods to implement the behaviour.

If you are too keen on deleting databases after test is done, then here is how Cosmos does it. (Since Emulator supports limited no of collections)
https://github.com/aspnet/EntityFrameworkCore/blob/9be0341cfc60a852b7c30da444b460702e0a055d/test/EFCore.Cosmos.FunctionalTests/TestUtilities/CosmosTestStore.cs#L137-L146

Yes, that's surely SQL Server specific. In Firebird, for example, database creation is instant.
Dropping in Dispose mean the database name needs to be unique per store (aka no sharing), because other tests might be running using the same database.
So, looks like some kind of "reference" counting or unique names is my only option here. :(

You can write a script outside of testing to run after build has passed to drop all databases.

No I can't. I would have to know the names of DBs somehow (Firebird does not have a "master" database).

Maybe you can reuse what I do for SQL CE?

@ErikEJ in case of CreateShared the database is left behind anyway, if I read your code correctly.

Note that leaving databases behind (or at least having the option to) can also sometimes be useful for inspecting the data after a problematic test has already executed.

Agree. I usually stop the test while it's running and inspect the database at that point.


Add link to documentation on selfdestruct warning

When adding a contract with selfdestruct, the CLI raises a warning explaining it is a security risk (see https://github.com/zeppelinos/zos-cli/pull/347). Add a link to the documentation in the message so users can go and check for more information.
If the documentation on that is not already written, then we need to write it as part of this issue.

Addressed by https://github.com/zeppelinos/zos/pull/228


[ARM]: Fix ARM What-If formatter

added missing parameter in method call
Description
Fixes #18720
Testing Guide
az deployment what-if ...
History Notes

[arm] az what-if: Fix output formatting

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

ARM

@shenglol Could you please help to review this PR?

@miqm Thanks for fixing this!


bug: fix docs

The  docs page at elfpy.delv.tech is down. The problem is that Vercel requires python 3.9 and we are >=3.10. We want to push towards the latest Python to stay on top of the latest typing capabilities, so we need a solution that does not use Vercel.
We should also switch the name to something like elf-simulations.delv.tech

Most of the docs are dynamically generated. The build process it self is fine, what is missing is an automated (in our CI) process for building & hosting them. We will be resolving this soon.

@engn33r docs are here for now: https://elf-simulations.readthedocs.io/en/latest/
We will update links in a follow-up PR soon.

Marking the issue as done -- will update links asap.


[Event Request] Codeunit 7033 Price Source - Cust. Price Gr. - procedure FillAdditionalFields - OnAfterFillAdditionalFields

Please add the following existing Event OnAfterFillAdditionalFields to procedure FillAdditionalFields in Codeunit 7033 Price Source - Cust. Price Gr.
local procedure FillAdditionalFields(var PriceSource: Record "Price Source")
    begin
        PriceSource.Description := CustomerPriceGroup.Description;
        PriceSource."Allow Invoice Disc." := CustomerPriceGroup."Allow Invoice Disc.";
        PriceSource."Allow Line Disc." := CustomerPriceGroup."Allow Line Disc.";
        PriceSource."Price Includes VAT" := CustomerPriceGroup."Price Includes VAT";
        PriceSource."VAT Bus. Posting Gr. (Price)" := CustomerPriceGroup."VAT Bus. Posting Gr. (Price)";

	OnAfterFillAdditionalFields(PriceSource,CustomerPriceGroup); <-- NEW EVENT
    end;

    [IntegrationEvent(false, false)]
    local procedure OnAfterFillAdditionalFields(var PriceSource: Record "Price Source"; CustomerPriceGroup: Record "Customer Price Group")
    begin
    end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 10050.


Add files for Mongo

Hello,
This Pull Request adds files for mongoDB users. As I've written in the repl.it guide, I don't provide support for mongoDB right now.
I'll merge this once I think I'm done working on it.

Merged this Pull Request. Locking it for now. If needed, create a new issue.


Replace Docker examples with curl examples

Discussion and motivations here: https://github.com/meilisearch/integration-guides/issues/145

Two examples to run MeiliSearch are present in our repository documentations:

in the README.md, in the Installation section.
in the CONTRIBUTING.md, in the Development Workflow section.

TODO: change both examples in README.md and CONTRIBUTING.md the same way as we have already done it in this PR https://github.com/meilisearch/meilisearch-python/pull/348

Hello @Ayush0431
thanks for your interest in this project :)
FYI, we prefer not assigning people to our issues because sometimes people ask to be assigned and never come back, which discourages the volunteer contributors to open a PR to fix this issue.
We will accept and merge the first PR that fixes correctly and well implements the issue following our contributing guidelines.
We are looking forward to reviewing your PR 🙂


Draft safety guidance for Exit this Page component

What
Draft a a safety content page which links from the Interruption Page in the component.
Why

to help users understand what will happen when they use the Exit this Page button
to help user stay safe online

Who needs to work on this
Calvin, Ciandelle
Who needs to review this
Katrina, David, Owen, Beeps
Done when

[x] Drafted guidance
[x] Reviewed guidance with team
[ ] Reviewed at community crit
[ ] Published (as part of component)

Decision made for the community consultation:

During the asyncronous consultation and at the hackathon, we provide a template of safety content in the pattern


Update workflows

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (8af55c3) to head (c5e9f54).
Report is 21 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #334   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           49        49
  Lines         1270      1270
  Branches        50        50
=========================================
  Hits          1270      1270

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update macOS fleet to 2023-10-20.

Also adds instructions for minting base boxes using one of the machines in the fleet rather than using a local machine.

The openmpi regression is probably fixed by https://github.com/microsoft/vcpkg/pull/34652. This should also resolve some of the other cascades.

Closed because by the time I had time to fix clang 15 it's already november


[Results] Improve stats for RTA PerfIndex metric

The current PerfIndex metric reports the average with respect to all the tasks composing the workload:
https://github.com/ARM-software/lisa/blob/master/libs/utils/results.py#L253
This could introduce big variances in case different tasks have quite different averages.
A better approach could be that to compute the average for each task independently.
A single metrics for the PerfIndex of an experiment can than be defined by ordering the per-task averages and comparing with respect to the same ordered list of the base case.
This should be enough to build a single index which better reports the performance index variation with respect to the base experiment.

Closing as PerfIndex is currently available per-task in the rta analysis


support filter on query result

Hi Developers,
I suggest is it possible to add filter base on search result.
sometimes, we find many tags.
could we have another level-2 text filter or type filter (marco/function) ?

@simon8233 Feature implemented in codequery4vscode version 0.8.0. Path filter support has been added. Please read the documentation for details on how to use this feature.


I need to get it done

As a [role]
I need [function]
So that [benefit]
Details and Assumptions
* [document what you know]

Acceptance Criteria
gherkin
Given [some context]
When [certain action is taken]
Then [the outcome of action is observed]

let me get it done


Bridge often doesn't receive transactions from the SDK

I don't know why or what causes this, but it gets progressively worse the longer the bridge runs. There's about a 50% chance that restarting the bridge fixes it until it falls over again.
Opening this to mostly remind myself to investigate, and to track the problem as it is now a theme.

Presumably we can make this @turt2live 's problem if we do #454


Added support of acoount TnOptions orders

I added support of account TnOptions orders because of LineOptionsOrders is deprecated and will stops to work soon.

@pashamesh Awesome Thanks!
Will review today


Create a proper thread-safe queue, in lieu of the Dequeue grabbed from Github

This is more important than ever - I've finally noticed an inconsistency in the wild in one of my simulator applications. Can't reproduce it yet.
It's also a weird workflow, but still, the concept works - just need to make it thread safe.


Remove unneeded dependency and increase timeout

Signed-off-by: Kevin Earls<EMAIL_ADDRESS>This should fix #947 by removing an unneeded dependency in the e2e-tests-examples2 target.  It also adds increases some timeouts to avoid some other intermittent failures we've seen, and adds logging to help better diagnose failures.

@jpkrohling  @objectiser @pavolloffay  @rubenvp8510  Please review


Add Block Detail page

cc @csillag @buberdds @lubej

I am merging this, since Luka is on vacation


Keyboard browse added to all photo pages (that I have and could test)

I added the 'J/space' and 'K' style navigation within a photo list page, complete with smooth transitions.
It's under the feature flag 'enable_keyboard_browse' if you'd like to try it out.

Also, meant to add: I didn't add it to the cameras or places list pages, as I don't have those hooked up and couldn't test it. It should work with no problem (might need the correct class name passed into the include) but I didn't want to commit without being able to test it.

I just pulled it as is, but you should be able to drop this in to inc_pagination.txt

On nevermind, I misunderstood. Anyway, this should probably just be on by default because it is good!

I just realized that my $(window).load() callback misbehaves if the images are coming from cache. My browser doesn't cache anything from my p-flickr so I didn't see it but Donald here at work pointed out that this something to watch out for.
There's a jQuery plugin built specifically for this but I didn't want to add another dependency on what is a should-be-simple feature.
I'll keep an eye on it and fix it when I get the time.


Record context menu

If you stay cursor on Record variable, table fields are shown and primary key fields are marked with asterix:

It would be nice if the field type FlowField and FlowFilter were marked likewise (exlamation mark, question mark or another character):

Hi! Thank you for your suggestion. We are working on focusing the activity in this repository on addressing issues related to the AL development tools.
As this issue appears to require changes across multiple areas of our product, we recommend that you post this on our Ideas forum at aka.ms/BCIdeas, or vote up the idea if its already there.
We are constantly monitoring top ideas and will consider them for a future release.


Fix docstring typos in MLP.__init__

Fixes a few minor typos in #358.

LGTM! Thank you for the fix.


Settings moved to Repository Providers

Updated GitLab and GitHub Enterprise.

[ ] needs updated screenshot client-id-and-secret.png @scopegate pretty please again?

The new text is accurate following the recent changes, it'd be great to get this live.
If getting an updated screenshot is a problem, can we merge and re-add the screenshot in a follow up?
/c @scopegate

Sorry to hold this one up folks — these screenshots actually look fine as far as I can tell, unless there's something I'm missing? 🤔
:shipit:

Thanks for clarifying @plaindocs — screenshot updated 👍🏻


docker build

Hi, I want to know how i'd be able to build my own docker img of this with custom modifications

go to https://github.com/openspeedtest/Docker-Image

But i need to build my own image

You can find the Docker file for building your own image and uploading it to Docker Hub or your own Docker registry on that repository.


QR Image White margins

Hi, How to do setting QR Image White margin?margin than Larger

Hi @shfee, the "margins" are defined as quiet zones in the QR code iso specification. They are are official part of each and every QR code.
Nevertheless, last week I was asked, how to remove them. So I added a switch to remove the margins.  Have a look at the issues: https://github.com/codebude/QRCoder/issues/25
If you want to make them larger, you could reverse the removement logic.
Another option would be, to place the ready rendered QR code image centered onto a slightly larger Bitmap.

Thank you!


style=color-scheme not updating on click, requires page reload

After clicking on theme toggler page refresh was required to update from  style="color-scheme: light;" to style="color-scheme: dark;"
  if (theme === "light" || theme === "dark") {
            document.documentElement.style.colorScheme = theme;
          }

might need to add an else statement here for other themes, but will test that later.

resolved in commit ebb2cb1