Automatic underlining of table and column names in sql editor

Description
Since update to Community Version <IP_ADDRESS>312241705 the SQL Editor formats table and column names in red underlined font. This happened for already opened scripts as well as for newly typed statements. The underlining appears and disappears sporadically during typing. Sorry that I have not identified a pattern how and when it occurs, but please see the below screenshot

If I add another identical line, the underlined table name in the last statement is formatted normal.
As it seems, this only happens when I connect to an Ingres DB, but not for Oracle, SQL Server, Access or Derby
DBeaver Version
Community Edition <IP_ADDRESS>312241705
Operating System
Windws 10
Database and driver
Ingres iijdbc-10.2-4.1.10
Steps to reproduce

Click on existing DB connection and open SQL Console
Type SQL Select statements with table and column names
End statement with semicolon or not and press enter
Type the same in the next column: table name will be underlined or not (I am not joking, but it is unfortunately not consistent)

Additional context
I know it is "only" a formatting issue, but this font is so annoying that I can't visually read my scripts anymore.
I also tried to fix it with SQL Editor Code formatters but that does not fix it. I have also already reset User preferences via Reset settings dialoge without success.

Such highlighting is the validation of tables and columns by metadata. It tells you that if you try executing your script, it will fail, and the reason is highlighted.
You can turn this highlighting off for the concrete database in the Preferences -> Editors -> SQL Editor -> Code Editor ->Read database tables/columns for semantic analysis.

Thanks a lot, that got rid of the underlined tokens and gave me back a great DB tool!
But then this validation is not correct, as the query runs very well if I end the statements with a semicilon. As you can see in my screenshot, I typed the same select 6 times but only 2 had underlined table names.
And why was it not there before the update? Did I miss something?

It's our new feature. Unfortunately, we don't have Ingres database in our test environment to check on our own. Could you help me to understand the roots of the issue? Does it get better if you put the delimiter ; to the end of every query?

For me that highlighting issue occurs when using synonyms (Oracle database). Using the fully qualified names (schema.table) works.

With Oracle I get it for views too (e.b. all_tables), but not for tables.
I tried to record what it looks like with Ingres (3 consecutive gifs), as the underlines appear and disappear as you type. Perhaps you can understand the underlying logic. What always seems to work is to also specify the scheme. However, it then doesn't recognize that the table x1100pattt doesn't even exist.

Let me know if you would like to see more for Ingres.

Thank you, I'll try to reproduce with Oracle.

When you don't put ; and don't add new line between queries we assume that it's not several separate queries. That's why validation doesn't work in this case. It's an expected behavior. Just separate queries with a new line or put ; at the and.
The issue which looks like a bug here is that your x1100pat table is not highlighted as table while you add schema name there.
Is dps chosen as an active schema for the current script? You can check it here:

The issue which looks like a bug here is that your x1100pat table is not highlighted as table while you add schema name there.

But it does present a list of available tables when I type the schema name followed by '.', so it looks like it would the meaning of each token.

Is dps chosen as an active schema for the current script? You can check it here

That's a good question: it never seemed to work for Ingres. I can see a list of the available schemas in the dropdown or the dialog, but when I select one, it stays at <N/A>. However, I can see that it works for Oracle or SQL Server and even updates when I switch schemas by typing in the console

I get the same issue with PostgreSQL. I have a single large query with multiple With statements, but the final SELECT (after the WITH statements) is made very difficult to read by every table or column referenced being in bright red, italic and underline. It is so annoying. DBeaver used to be a joy to use till the recent update made reading SQL that much harder.

Thanks, but would I lose any valuable analytical functionality? Allowing us to alter the colour of the highlighted tables/columns would be a great help, while retaining the functionality.

@ctopenpetra you can turn it off #22320 (comment)

Thanks, but would I lose any valuable analytical functionality? Allowing us to alter the colour of the highlighted tables/columns would be a great help, while retaining the functionality.
On a new SQL script tab for my PostgreSQL database, I type the simplest of queries and get:
SELECT
*
FROM
account a
WHERE
a.account_code IN ('ABC', 'CDE');
For no reason, account_code  in the WHERE clause is bright red, italics and underline.

@ctopenpetra thank you for sharing your case.
Unchecking Read database tables/columns for semantic analysis will turn off only query validation by metadata. Seems it has some bug which we're going to fix soon.
Syntax structure validation will work - for example, if you reference table in select, which wasn't specified in from clause.

@ctopenpetra thank you for sharing your case.
Unchecking Read database tables/columns for semantic analysis will turn off only query validation by metadata. Seems it has some bug which we're going to fix soon.
Syntax structure validation will work - for example, if you reference table in select, which wasn't specified in from clause.

Thanks, Elizabeth. That works perfectly.

I would like to add to this issue, that, when editing a stored procedure, all table columns are in that bold red highlight font because the connection shown in the main toolbar of the current editor window (sp-editor) is "N/A".
For some reason, i never see the active connection up there, even when i double-click a stored procedure.
but without connection, the editor thinks, all symantic checks fail and therefore highlights everything in this eye-burning red.
for now, only option is to deactivate semantic analysis in an sp-heavy project.

I have the same issue on PostgreSQL. Even simple queries show a lot of errors, even though the command runs fine.
I've disabled the highlighting, but it would be a really nice feature if it worked.

I'll echo the previous comment. If you're going to mark something as an error, it really should be an error. This feature is flagging semantically correct statements that use different case in a language that is mostly case-insensitive. It seems like a useful feature, otherwise, so hopefully that can be fixed soon.

Thanks. 😊
Jack Goldstein
SOS Compliance Reporting Team
https://w3.sos.ibm.com
From: Elizabeth @.>
Sent: Monday, January 15, 2024 09:54
To: dbeaver/dbeaver @.>
Cc: Jack Goldstein @.>; Mention @.>
Subject: [EXTERNAL] Re: [dbeaver/dbeaver] Automatic underlining of table and column names in sql editor (Issue #22320)
@jackgtor we're working on it — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: <dbeaver/dbeaver/issues/22320/1892322243@ github. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious  https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!2W-rSPTVYnMy-c8xkittRBc6Ix7gIK7zWCYGo-AO8Zb8aCiLiu2dQPreGC3i4XRwrpFhPWjXkBmBgAMJZRqCTu_qCFs8vrRTt8Ac$
ZjQcmQRYFpfptBannerEnd
@jackgtorhttps://github.com/jackgtor we're working on it
—
Reply to this email directly, view it on GitHubhttps://github.com/dbeaver/dbeaver/issues/22320#issuecomment-1892322243, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BFLNN6XCYFYGTQN5Q4JEV33YOU7HFAVCNFSM6AAAAABBMBKNOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJSGMZDEMRUGM.
You are receiving this because you were mentioned.Message ID: @.***>

Such highlighting is the validation of tables and columns by metadata. It tells you that if you try executing your script, it will fail, and the reason is highlighted. You can turn this highlighting off for the concrete database in the Preferences -> Editors -> SQL Editor -> Code Editor ->Read database tables/columns for semantic analysis.

This does not work as intended! This feature now highlights fields in queries that are perfectly fine, see example below. In addition, it really makes it hard to read field names with underline highlighting, when the field name has underscore in it.

We made some fixes and they're available in the Early Access version. You could try https://dbeaver.io/files/ea/

Hi, Is it possible to switch from the stable version to early access and vice versa ?
Best regards.

@m3n3chm0 yes, just run the installer of the version you want and the version will change, while connections and settings remain

Ok, @E1izabeth. Really appreciated!

It's much better now. Thanks for the quick fix.
I have one more question which may or may not be related.
When connected to DB2 UDB, it's not recognizing current_date or current_timestamp as valid SQL in the sense that it's not auto-completing them and is highlighting them in red underline. The expressions still run properly. When connected to PostgreSQL, they are identified correctly. From where is it getting the list of recognized SQL words? Is that a list I can customize?
Thanks,
Jack

@jackgtor could you create a separate issue about adding current_date and current_timestamp keywords for DB2?

Thanks Elizabeth. Feature 22563 has been opened.

Hi, as others have said this is working better now. I'm still seeing an issue where the metadata validation does not follow aliases (using DB2 for LUW). If the query references a table directly it works well, but if an alias is used (which I personally use extensively) it always gets highlighted as an error even if the table and column names are valid. Is this expected to work?

Thanks Elizabeth. It's significantly better for my Oracle DB queries.
However for Snowflake DB it works strange, but might not directly because of this feature.
If I use all uppercase, it can't find the columns. I need to use lowercase or quote the table name. However I can use lower and upper case schema names.

The strange thing why I mentioned it's might related to other issue, is when I use the built in auto completion the table name sometimes became quoted, sometimes doesn't. The incorrect underlining issue only persist when the autocomplete would quote the table name.

@TB-HunBoka
Thank you for feedback. I can reproduce it, but only with VIEWS view, other views and tables works fine for me. Do you experience this issue only with VIEWS?

@mtmcg could you provide some example?

Hello @E1izabeth
No, I experienced the same problem with a single table, and couple of views.
However, I think I found the problem.

When an object has the same name as an sql word or command, currently it will be quoted by the autocomplete function and this underlining function also requires it to be quoted, otherwise it will not scan the object for columns.

@E1izabeth
Yes I will.
In the mean time I found another problem with the current version of this function.
The Having command validation for currently selected column names only. Furthermore the selected column name and the condition in the having command needs to be directly selected and not only aggregated:

@E1izabeth You were right, the having part is now validated correctly in the early version. Thank you for the fix.

I'm closing this issue as it seems resolved.
If you find some inconsistency, please feel free to create a new issue.

Then when I enable Read database in Query Analysis, I get this:

Completely lose SQL column name foreground.
That's with the early access.
First day trying the app.

@rasqual please create a new issue


[$250] Android - Emoji isn't fully visible after editing - reported by @thesahindia

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open any chat
Comment a single emoji
Edit the comment and change the emoji .

Expected Result:
Changing emoji should work fine.
Actual Result:
Emoji is not fully visible.
Workaround:
unknown
Platform:

Where is this issue occurring?

Android

Version Number: 1.1.44-4
Reproducible in staging?: Y
Reproducible in production?: Y
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/43995119/159848510-9180cc74-6565-492e-af5b-0bca3a68a045.mov
Expensify/Expensify Issue URL:
Issue reported by: @thesahindia
Slack conversation: https://expensify.slack.com/archives/C01GTK53T8Q/p1647287252898019
View all open jobs on GitHub
this job post

Testing:

iOS 15.3.1
NewDot version 1.1.44-4

Sending to eng!

@yuwenmemon Whoops! This issue is 2 days overdue. Let's get this updated quick!

Sending to eng - not sure if it's related to this? https://github.com/Expensify/App/issues/8115

Don't think so. This can be worked on externallty

I'm not sure why <Text style={styles.w1}>{' '}</Text> was there in the first place, but getting rid of it does resolve this issue. Let's go with @dharmik's proposal # 2, but let's additionally remove the comment above it as well.

Agreed

@rushatgabhane @deetergp Applied on upwork. ✅

@rushatgabhane @deetergp @jboniface Applied on upwork. ✅

@deetergp @rushatgabhane https://github.com/Expensify/App/pull/8548 PR created.

@deetergp @rushatgabhane https://github.com/Expensify/App/pull/8585 PR created.

The PR for this has bee submitted and merged.

@deetergp Is there any something pending?
CC @jboniface

@jboniface Is something for this task?

looks like the automation that alerts us to pay failed. I paid the contract for you @dharmik .
That said, @thesahindia and @rushatgabhane need to apply to get paid for their parts as well. The other posting expired, so you both can use this one.

Done ✅

done

@jboniface we can close this out after the contract is closed :)

done!


Switch node signaling mechanism to use threading.Event()

Currently, node signaling is done using locks. We can simplify the signaling mechanism by switching to threading.Event()

done within 2 days.


post.summarize for a user-defined excerpt

To allow for an excerpt, I used to use a chyrp plugin which chopped a post wherever you put a "" html comment marker to get just an excerpt (i.e the first section) of the post. It should be easy to parameterise the marker text if you like, but I've provided the above implementation here:
http://github.com/gfxmonk/jekyll/commit/8cc666b53a83c33d80785a3a049f0a454aacf3bd

@mojombo Can we have this?


Can't add extensions

Steps to reproduce

Open the extensions tab

Expected behavior
View a list of extensions
Actual behavior
ಥ‿ಥ
"that's strange"

Crash logs
No response
Mihon version
App version: 0.16.2
Android version
Android 12
Device
Motorola g60
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have gone through the FAQ and troubleshooting guide.
[X] I have updated the app to version 0.16.2.
[X] I have updated all installed extensions.
[X] I will fill out all of the requested information in this form.

There are no first part extensions

search tachiyomi extensions in google


Fix 2.1 docs

Fix #2345
Update docs to talk about dotnet 8 and yarp 2.1.0

Since this is just doc fixes it can go directly into the 'latest' branch. Make sure it gets into main as well.

I think it's important to fix the 2.1 branch, then port it to latest and main: in the future, if someone wants to look at the 2.1 doc, it will be correct.


[HOW-TO]

Is there a way to display the preview window with Tkinter? In PiCamera I was able to place a preview window independently of TKinter. I don't want to use QT in my project.

I would say try with a short code example and if it's solid enough you can share it in examples folder via pull request. I'm working too on an option to take pictures by pressing a key and stopping the script with a key (without OpenCV).

Here is the process "camera" from my project showing a preview window and capturing. I would like to create the same with Picamera2, but without CV2 and QT, only with Tkinter.
#importing required modules
import logging
import logging.handlers
import multiprocessing
import picamera
from pathlib import Path
import sys
#importing required custom modules
from Model.constants import *
#camera child process (overriding "def run" method)
def camera(control, control_queues):
#logger
queue_handler = logging.handlers.QueueHandler(control_queues[LOGGING])

#root logger for this child process
root_camera_logger = logging.getLogger()
root_camera_logger.setLevel(logging.DEBUG)
root_camera_logger.addHandler(queue_handler)

#logger for this child process
camera_logger = logging.getLogger("camera logger")
camera_logger.log(logging.INFO, "process started: %s" % multiprocessing.current_process().name)

#multiprocessing logger
camera_mp_logger = multiprocessing.get_logger()
camera_mp_logger.setLevel(logging.DEBUG)
camera_mp_logger.addHandler(queue_handler)

try:

    #init picamera
    cam = picamera.PiCamera()

    #place the preview window independent of the tkinter GUI
    cam.preview_fullscreen = False
    cam.preview_window = (3, 17, 295, 295)
    cam.resolution = (295, 295)

    #start endless loop for the camera process
    while not control[QUIT_PROCESS]:

        if control[CAMERA]:

            if control[TOGGLE_CAMERA]:
                cam.start_preview()
                control[TOGGLE_CAMERA] = False

            if control[CAPTURE]:
                try:
                    cam.capture(str(Path(sys.path[0] + "/Media/Datamatrix/TEST.PNG"))) #DTMX.PNG
                    control[CAPTURE] = False

                except IOError as err:
                    camera_logger.log(logging.DEBUG, "cannot write captured datamatrix code! {0}".format(err))
                    control[CAPTURE] = False

        else:
            cam.stop_preview()

    camera_logger.log(logging.INFO, "camera process closed")

except Exception as e:
    template = "An exception of type {0} in process 'camera' occurred. Arguments:\n{1!r}"
    message = template.format(type(e).__name__, e.args)
    print(message)

@bnd762 I would try to replace everything where you used PyQt and/or OpenCV and find the euqivalent Tkinter functions.

In legacy OS images it was possible to render camera images to the screen directly using the HDMI hardware, but the desktop "knew" nothing about the images that were simply written on top of it. In more recent versions of the OS you need to go through some existing Linux display framework and the old mechanism is no longer supported.
I'm afraid I don't know anything about TkInter, though I expect it must be possible to render camera images that you capture with it (GPU acceleration may be trickier). Sorry not to be more help.

I could pick up the individual images via Mapped Array and display them in Tkinter to simulate a preview. But that's slow

In the Qt world you can use software rather than hardware-assisted rendering, and on a Pi 4, so long as you feed it an appropriate image format, and the resolutions aren't too big, it's not terrible. You could try it and see - presumably that's the performance level you could achieve without delving into OpenGL. The camera images are in uncached memory currently, so the best policy is often just to take an immediate copy and work with that.

There must be a way to do this in Python/Tkinter without QT and CV2.
It can't be that it doesn't work.

I have now solved the problem with pygame and it works wonderfully. I use multiprocessing. Pygame runs in a separate process and is only used for the camera. The GUI (tkinter) runs in the main process. In the code example I have solved the problem that the Pygame window is in the foreground on the Raspberry X11 and under Windows.
#importing required modules
import logging
import logging.handlers
import multiprocessing
from picamera2 import Picamera2
from pathlib import Path
import sys
import os
#hide support prompt message
os.environ['PYGAME_HIDE_SUPPORT_PROMPT'] = "hide"
os.environ['LIBCAMERA_LOG_LEVEL'] = "3"
import pygame
#windows only
if sys.platform.startswith("win"):
from ctypes import windll

#importing required custom modules
from Model.constants import *
#camera child process (overriding "def run" method)
def camera(control, events, control_queues):
#helpers

#logger
queue_handler = logging.handlers.QueueHandler(control_queues[LOGGING])

#root logger for this child process
root_camera_logger = logging.getLogger()
root_camera_logger.setLevel(logging.DEBUG)
root_camera_logger.addHandler(queue_handler)

#logger for this child process
camera2_logger = logging.getLogger("picamera2")
camera2_logger.setLevel(logging.INFO)

camera_logger = logging.getLogger("camera logger")
camera_logger.log(logging.INFO, "process started: %s" % multiprocessing.current_process().name)

#multiprocessing logger
camera_mp_logger = multiprocessing.get_logger()
camera_mp_logger.setLevel(logging.DEBUG)
camera_mp_logger.addHandler(queue_handler)

try:

    #init picamera##https://github.com/raspberrypi/picamera2/issues/527
    cam = Picamera2()

    #place the preview window independent of the tkinter GUI
    os.environ['SDL_VIDEO_WINDOW_POS'] = "%d,%d" % (30,17)

    #X11 raspberry only -> set window always on top -> no function!?
    os.environ['SDL_WINDOW_ALWAYS_ON_TOP'] = 'SDL_TRUE'

    #windows only
    if sys.platform.startswith("win"):

        SetWindowPos = windll.user32.SetWindowPos

    res = (240, 320) #295, 295

    cam.preview_configuration.main.size = res
    cam.preview_configuration.main.format = 'BGR888'
    cam.configure("preview")

    #configure still capture
    capture_config = cam.create_still_configuration()
    cam.still_configuration.main.size = (2500,2500)

    #start endless loop for the camera process
    while not events[QUIT_PROCESS].is_set():

        if control[CAMERA]:

            if control[TOGGLE_CAMERA]:

                pygame.init()

                cam.start()

                #borderless window
                screen = pygame.display.set_mode(res, pygame.NOFRAME)

                #windows only -> set window always on top
                if sys.platform.startswith("win"):

                    SetWindowPos(pygame.display.get_wm_info()['window'], -1, 3, 17, 0, 0, 0x0001)

                #raspberry X11 only -> set window always on top
                #sudo apt-get install wmctrl
                else:

                    os.system("wmctrl -r 'pygame window' -b add,above")

                control[TOGGLE_CAMERA] = False

            if control[CAPTURE]:######control_events[CAPTURE].wait()

                try:

                    #cam.capture(str(Path(sys.path[0] + "/Media/Datamatrix/TEST.PNG"))) #DTMX.PNG
                    cam.switch_mode_and_capture_file(capture_config, str(Path(sys.path[0] + "/Media/Datamatrix/TEST.PNG"))) #DTMX.PNG
                    control[CAPTURE] = False######control_events[CAPTURE].clear()

                except IOError as err:

                    camera_logger.log(logging.DEBUG, "cannot write captured datamatrix code! {0}".format(err))
                    control[CAPTURE] = False######control_events[CAPTURE].clear()

            array = cam.capture_array()
            img = pygame.image.frombuffer(array.data, res, 'RGB')
            screen.blit(img, (0, 0))
            pygame.display.update()

        else:

            #to prevent error check if cam is running
            if cam.started:

                cam.stop()

                pygame.quit()

    camera_logger.log(logging.INFO, "camera process closed")

except Exception as e:

    template = "An exception of type {0} in process 'camera' occurred. Arguments:\n{1!r}"
    message = template.format(type(e).__name__, e.args)
    print(message)

Cool, glad it's working. It might be nice just to put a display-a-camera-image-using-pygame example into the examples folder too!


Need separate development and production requirements in pipenv

Rob Rodrigues correctly noted that the project does not separate the development and production requirements in pipenv.
Fixing this is good pipenv hygiene.

Fixed in v0.3.3.


MM-43816_Unable to hit enter to create channel in new Create Channel modal

Summary
New channel modal: enabling enter key to submit channel creation
Ticket Link
https://mattermost.atlassian.net/browse/MM-43816
Screenshots

Release Note
NONE

/e2e-test

Successfully triggered e2e testing!
https://git.internal.mattermost.com/qa/cypress-ui-automation/-/pipelines/191015

One issue with placing this event on the modal rather than on the input is that you can't hit enter in the textarea (for the purpose). Not sure if this is expected?

Just checked with UX and agreed to disable the submit from the Purpose textarea. Updated!

/cherry-pick release-6.7

Cherry pick is scheduled.

Cherry pick is scheduled.

/cherry-pick cloud

/cherry-pick release-6.7

Cherry pick is scheduled.

/cherry-pick cloud

Cherry pick is scheduled.


Functional component not working

Environment

Operating System: Windows_NT
Node Version:     v16.13.1
Nuxt Version:     3.2.3
Nitro Version:    2.2.3
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      extends, ssr, routeRules, runtimeConfig, css, i18n, modules, alias, vite
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
https://stackblitz.com/edit/nuxt-starter-gnsutb?file=nuxt.config.ts
Describe the bug
It seems that the <i18n> functional component from vue-i18n (https://kazupon.github.io/vue-i18n/api/#i18n-functional-component) is not working with this module.
Additional context
No response
Logs
No response

Thank you for your reporting!
i18n component has deprecated from vue-i18n v9.
you need to renane to i18n-t from i18n
breaking changeing docs is here
https://vue-i18n.intlify.dev/guide/migration/breaking.html#rename-to-i18n-tfrom-i18n
nuxt i18n module is using that version.
Thanks!

Thank you for your reporting! i18n component has deprecated from vue-i18n v9. you need to renane to i18n-t from i18n breaking changeing docs is here https://vue-i18n.intlify.dev/guide/migration/breaking.html#rename-to-i18n-tfrom-i18n
nuxt i18n module is using that version.
Thanks!

thanks a lot, I missed it! But then, the docs should be saying that it's deprecated here: https://kazupon.github.io/vue-i18n/api/#i18n-functional-component !!


rough demo of fullscreen working

Hello @rejas
This is a very rough cut of fullscreen API implementation. Can I get your opinion and thoughts on how you'd like to see it proceed?
ATM, I turned off single click listener on image. Doubleclicking will cause the wrapper to be made fullscreen.

I like the idea very much, it's simple, doesnt rely on 3rd libraries and progressivly enhances when you use a modern browser.
The question on how to invoke fullscreen is also important for touch enabled devices: is dbl-click the same as double-tap in jquery? and whats the reasoning behind the single-click-turned-off ?

Thank you!

whats the reasoning behind the single-click-turned-off

This is just for testing purposes. I need to figure out how to properly allow for single clicks and double clicks. This will take a bit more research as, I think, it will require some way to measure the time difference between clicks/taps.

invoke fullscreen is also important for touch enabled devices

Yes! This will need to be tested for. And I definitely agree this needs to be implemented, as you say, as a progressive enhancement .
Thanks again for your input.


Check for magic number in headers

This change checks for the magic number 9994 in the first 4 bytes of the header as per the shapefile spec (https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf). This is useful for detecting a non-shapefile being parsed as though it were a shapefile.

Codecov Report

Merging #38 into master will decrease coverage by 0.07%.
The diff coverage is 64.28%.

@@            Coverage Diff             @@
##           master      #38      +/-   ##
==========================================
- Coverage   60.04%   59.97%   -0.08%
==========================================
  Files           7        7
  Lines         801      812      +11
==========================================
+ Hits          481      487       +6
- Misses        264      267       +3
- Partials       56       58       +2

Impacted Files
Coverage Δ

sequentialreader.go
50.42% <50%> (-0.46%)
:arrow_down:

reader.go
61.31% <75%> (+0.24%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c721ed4...2b9a917. Read the comment docs.

@fawick , I've address your suggestions, please let me know if there are any extra changes you'd like before merging.


Better English UI strings, minor cleanups

Created a new PR based on #5323 since it was hard to merge (files moved in the meantime).

Nicely done! :)

Could you guys please go over EN on https://hosted.weblate.org/projects/evcc/evcc/ to finalise it there ?


Could not find a declaration file for module

Not sure if this is reproducable, but I encountered the following error:
vite.config.ts:10:27 - error TS7016: Could not find a declaration file for module 'vite-plugin-remove-console'<EMAIL_ADDRESS>implicitly has an 'any' type.
  Try `npm i --save-dev @types/vite-plugin-remove-console` if it exists or add a new declaration (.d.ts) file containing `declare module 'vite-plugin-remove-console';`

10 import removeConsole from 'vite-plugin-remove-console'
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Found 1 error in vite.config.ts:10

I think it's your project's problem, please delete node-modules and pnpm-lock.yaml and reinstall dependencies

refer https://github.com/xiaoxian521/vite-plugin-remove-console/tree/main/playgrounds

hey, upgrading to the latest version fixes this


"Couldn't alloc class applekeystoretest" when installing Ventura

Which operating system are you trying to install?
e.g. macOS Ventura
Describe the bug
When i try to simple open the BaseSystem file, it runs to certain point that gives me that error: "couldn't alloc class applekeystoretest"
To Reproduce
Steps to reproduce the behavior:

Select and click baseSimstem in  OpenCore and go directly to the error.

Expected behavior
Go through the verbose until start the recovery mode
Desktop (please complete the following information):

OS: ArchLInux
Linux archlinux 6.2.10-arch1-1
Ryzen 7 5700g

In my reasearch:
"-cpu Haswell,vendor=GenuineIntel" is a must, i dont know if this could cause performance issues....
" -smp cores=4,threads=2,sockets=1" - more than that the BaseSystem wont open

@kingsvell for the cpu haswell part you can change it to cpu host

I have the same issue but using one core and one thread allows me to go through the installation, which doesn't really help because the performances are horrible...

Recovery gets locked up here on 1core


Enable updating the language of artefacts

This should help when content is incorrectly marked as english, but
isn't actually english. I see no reason to restrict this in the
interface, I'm guessing it just wasn't considered when trying to merge
Panopticon in to Publisher.
:warning: This application is Continuously Deployed: :warning:

Merged changes are automatically deployed to staging and production.

Make sure you follow the guidance for deployments before you merge.

Check your branch is being deployed in the Release app, after merging.

This'll require https://github.com/alphagov/publishing-api/pull/1849 to work reliably (or maybe at all).


Add docs to readthedocs.org

is there any wiki or docs that we can collaborate . i think is ipperative to mention in case of using database it must have a session table that can minphp  store sessions , a chemaforthis
CREATE TABLE sessions (
id varchar(64) COLLATE utf8_unicode_ci NOT NULL,
expire datetime NOT NULL,
value text COLLATE utf8_unicode_ci,
PRIMARY KEY (id),
KEY expire (expire)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

The SQL is actually available in /components/session/session.sql. However, minphp is going through a major refactor for version 1.0, as well as 2.0, so changes are coming!

can i imagine namespace will be sued in next major release ?

also , the next major release need our own code to be changed also ? i'm working now all my project with minphp . so have you any idea tips to make my code easy or compatible with next releases ?

The next version of minphp (1.0.0) will use namespaces, but will be 100% backwards compatible with the current version (0.x). We're able to do that by bridging classes from the global namespace to specific namespaces.
So you won't need to make any changes, but there will be some functionality deprecated in 1.0.0, that will be removed in 2.0.0, since 2.0.0 won't have any classes in the global namespace.

so have you any idea tips to make my code easy or compatible with next releases ?

Just use PSR-2 formatting ;)


emoji's won't wrap at the end of a line unless there is extra text after the line

This is in Chrome on a Mac in PTT 2.2.1
I've defined a PTT width that should wrap at the dotted blue line

As you can see it doesn't. My text there is "Love all, trust a few, do wrong to none. 😇🇨🇽🎅🏻". However, if I append text to the end of the string like this: "Love all, trust a few, do wrong to none. 😇🇨🇽🎅🏻 this"

Then the text wraps properly at the emoji.

I'll try to reproduce this. but if it's easy for you to check, can you see whether

adding just a space or newline after the emojis makes it wrap?
If this happens to other text besides emojis?

Sorry for the delay. Adding a space after the emoji is actually worse as the emoji disappears.

And yes, it appears if I add just 1 word where there is supposed to be a wrap it doesn't wrap. Likewise if I add a space after that word it causes the word to disappear.
So ultimately I guess this isn't about emoji's but something to do with the wrapping code.

I'm also experiencing this issue! When wordWrap is set to true, if the last line should have exactly 1 word, it instead is included in the previous line.

I'm working on this and #118 at the same time. Both are attempts to fix the wrapping behavior of the last word in a line when it's close to the wrap width.

To simplify the problem, when the last word in the text pushes the width of the line its on > than the word wrap width, the word should wrap to the next line but currently does NOT wrap (it just keeps going)

Can you guys check out  v2.2.7? This should be working correctly in that version. Let me know if there are still issues.

That issue appears to be fixed, but I'm now seeing some issues where an extra space is being added at the beginning of certain lines.  It looks like it's happening in cases where the previous line is especially long, possibly just when the previous line is exactly at the wordWrapWidth.

If you could give me some testable code for your text field that would be
great
On Sat, Jul 24, 2021, 20:30 apgiorgio @.***> wrote:

That issue appears to be fixed, but I'm now seeing some issues where an
extra space is being added at the beginning of certain lines. It looks like
it's happening in cases where the previous line is especially long,
possibly just when the previous line is exactly at the wordWrapWidth.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/mimshwright/pixi-tagged-text/issues/100#issuecomment-886141482,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABCU2EV2FBUVAS4Z3GV7YLTZOANDANCNFSM47VLGEZA
.

Also @apgiorgio i don't know if it was a randomly generated one but your avatar looks like a pink skull with sunglasses which is awesome!

Okay I managed to recreate it in a reproducible way!
const multiStyles = {
                "default": {
                    fontFamily: 'arial',
                    fontSize: 12,
                    fontWeight: 'bold',
                    fill: 0x333333,
                    align: 'left',
                    wordWrap: true,
                    wordWrapWidth: 200
                },
                "kw": {
                    fontFamily: 'arial',
                    fontSize: 12,
                    fill: 0xE24747,
                    fontWeight: 'bold',
                    align: 'left'
                }
            };

let description = new TaggedText("When played, give an allied troop <kw>Summoned</kw>, then sacrifice it.", multiStyles);

For me, this ends up displaying the following:

@apgiorgio 2.2.8 should fix it!
Redid/simplified the whole layout sequence.

Yeah that looks great, thanks!


replace json.load-example in README.md

Great library!
The example in README.md for opening an OAS-spec-file doesn't work:
oas = json.load(<url_to_specification>)

Error:
AttributeError: 'str' object has no attribute 'read'

Proposal:
from urllib.request import urlopen

with urlopen('<url_to_spec>') as spec_file:
    oas = json.load(spec_file)

Thank for pointing out the bug in the example. I changed the faulty part to:
import yaml
import requests
...
url = ("https://raw.githubusercontent.com/"
      "OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml"
      )
oas = yaml.safe_load(requests.get(url).text)


Redis operation information cannot be displayed in tracking

Please answer these questions before submitting your issue.

Why do you submit this issue?
[x] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

I want to see the specific operation of redis in the trace, but the component can't recognize it all the time. What it shows is: Spring MVC.Jedis2. X for redis link library

端点:com.zcsy.testapm.controller.APMTestController.testRedis()
跨度类型:Entry
组件:SpringMVC
Peer:No Peer
失败:false
url:http://<IP_ADDRESS>:28081/testapm/redis
http.method:GET

This is the SpringMVC span, not redis' span


Inconsistent tab order / initial focus

Bug Description
I have been reviewing the tab order for keyboard/screen-reader users, and it appears that sometimes focus starts in the top left and follows a logical order top-to-bottom and left-to-right and other times it jumps straight into the simulator pane. Once underway the tab order is consistent, it's just like we've short-circuited it by focussing the simulator at an inopportune moment.
How To Reproduce
Sometimes I can reproduce this consistently by using a screen reader, or not. But I imagine this is purely coincidentally, since the page should not know whether a screen reader is being used. And it doesn't always occur consistently.
Steps to reproduce the behavior:

Load https://python.microbit.org/v/beta
Hit F5 to refresh to get consistent starting state
When page is loaded, press tab a few times and observe focussed elements

On several occasions I have seen the correct behaviour occurring when either Narrator or NVDA are running, and the incorrect behaviour occurring when they are not. As I started writing this issue I was observing the bug on the live editor, but right now I can only trigger it on a local version (2fd2bfd).
Expected behaviour
Tab order should start from top left like in screenshot below
Screenshots
Correct tab order
Note that there may be a separate discussion to be had about the best tab order, depending on whether the left hand pane is collapsed or expanded, but this certainly seems to be the intended behaviour.

Incorrect tab order

Environment

OS: Windows 10
Browser: Chrome (105.0.5195.127) and Firefox (104.0.2)
Version: Editor 0.1.0-dev.3084
MicroPython (micro:bit V1) 1.1.0-beta.1
MicroPython (micro:bit V2) 2.1.0-beta.3

It is likely that this issue was fixed by https://github.com/microbit-foundation/python-editor-v3/pull/900 merged as ac0a9ec963090f3e236ea50e2b88651fd2502c84 on September 8th vs your local commit from end of August.

@microbit-robert good spot that a soft/hard refresh may give different behaviour with NVDA - presumably it maintains some state on a soft refresh. The issues you describe are separate to the very specific behaviour I described above (but am struggling to replicate now, so could have been fixed by ac0a9ec). I'll open a separate issue to discuss the overall tab order concerns.

Reading around this, NVDA by design remembers where the user's last position when returning to a previously visited web page. This isn't working quite as expected with the Python Editor.

Somewhat more inexplicably, running the Editor locally yields different behaviour to the beta. Locally, I always start focussed in the simulator.

Somewhat more inexplicably, running the Editor locally yields different behaviour to the beta. Locally, I always start focussed in the simulator.

Does this occur with and without NVDA running?

Somewhat more inexplicably, running the Editor locally yields different behaviour to the beta. Locally, I always start focussed in the simulator.

Does this occur with and without NVDA running?

This is without NVDA, but I suspect introduced by https://github.com/microbit-foundation/python-editor-v3/commit/ac0a9ec963090f3e236ea50e2b88651fd2502c84.

I suspect if you run NVDA you will "fix" it, could you please test?

#982 has fixed all issues relating to focus and tab order with and without NVDA when the project is run locally (not inside an iFrame). When using the beta deployment (inside an iFrame), NVDA is causing the focus to switch to different areas of the Editor.

All being well with some other tasks, we'll remove the iframe for the Python Editor launch. However, we still plan to embed the app in the micro:bit classroom so it's worth us exploring it further in that context.

All being well with some other tasks, we'll remove the iframe for the Python Editor launch. However, we still plan to embed the app in micro:bit classroom so it's worth us exploring it further in that context.

This issue should now be fixed for the beta deployment.
The micro:bit classroom scenario is affected by this issue (as it embeds the editor in an iframe) and we've raised an internal ticket for this.


Handle requests with multiple Accept headers

RFC 7230 HTTP/1.1 Message Syntax and Routing Section 3.2.2.  Field Order
states that a server MAY combine multiple header fields with the same
name into a comma-separated list [i.e. #(values)], withouth changing the
request semantics.
This change transforms multiple Accept headers in a request into such a
comma-separated list.

Note that all Travis tests are failing for this change.

Note that all Travis tests are failing for this change.

Everything works for me locally (except the deflate_SUITE see #315).
See e.g. https://travis-ci.org/klacke/yaws/jobs/306288909#L1572
It seems Travis runs great until it hits the revproxy_SUITE, where it fails with badbind:
  revproxy_SUITE...

2017-11-23 12:21:10 crash_report

    initial_call:     pid:     registered_name:     error_info:     ancestors:     messages:     links:     dictionary:     trap_exit:     status:     heap_size:     stack_size:     reductions: 2017-11-23 12:21:10 ** Generic server ~p terminating

** Last message in was ~p~n** When Server state == ~p~n** Reason for termination == ~n** ~p~n

				2017-11-23 12:21:10 crash_report        2017-11-23 12:21:10 Error in process ~p with exit value:~n~p~n

		2017-11-23 12:21:10 crash_report

    initial_call:     pid:     registered_name:     error_info:     ancestors:     messages:     links:     dictionary:     trap_exit:     status:     heap_size:     stack_size:     reductions: 2017-11-23 12:21:10 Error in process ~p with exit value:~n~p~n

		Kernel pid terminated (application_controller) ({application_start_failure,yaws,{{shutdown,{failed_to_start_child,yaws_server,{badbind,[{yaws_server,start_group,2,[{file,"yaws_server.erl"},{line,261}]

Crash dump is being written to: erl_crash.dump...done

============================================================================

 see /home/travis/build/klacke/yaws/testsuite/logs/yaws.log for details

 common_test result<EMAIL_ADDRESS>
============================================================================

make[2]: *** [common_test] Error 1

make[2]: Leaving directory `/home/travis/build/klacke/yaws/testsuite'

make[1]: *** [check-am] Error 2

make[1]: Leaving directory `/home/travis/build/klacke/yaws/testsuite'

make: *** [check-recursive] Error 1

The command "make check -j4 V=1" exited with 2.
after_script

0.01s$ ./scripts/upload-travis-build-artifacts

ERROR; missing github token

Done. Your build exited with 1.

I would be surprised if this was due to my pull request.
I'm happy to help debug the CI issue!

There is definitely an issue with the testsuite on Travis. I'm unable to reproduce the problem on my own. So it is not really easy to fix it. But, I agree with @avtobiff, this is not a problem with the pull request.
BTW, I will try to review the PR very soon. Thanks !


Wrong url hint on Chrome Android for password manager autofill

Bug description
On the Portainer login screen on Chrome Android, when trying to use a password manager with autofill (like Bitwarden), the hint provided to the password manager is "--" instead of the current url, which prevents the password manager to find the correct entry.
Expected behavior
The password manager should receive the proper url to find the correct entry.
Steps to reproduce the issue:

Go to your portainer login page
Focus on the login or password field, the password manager autofill popup should appear
Click on the password manager autofill popup
The password manager opens, and search for entries with the url "--"

Technical details:

Portainer version: 1.23.0
Docker version (managed by Portainer): 19.03.1
Platform (windows/linux): Linux
Command used to start Portainer (docker run -p 9000:9000 portainer/portainer): Docker Compose, see additional context
Browser: Android Chrome 79.0.3945.93

Additional context
Docker compose to start Portainer:
portainer:
  image: portainer/portainer
  container_name: portainer
  ports:
    - 9000:9000
  volumes:
    - ~/opt/appdata/portainer:/data
    - /var/run/docker.sock:/var/run/docker.sock
  restart: unless-stopped

I will close this as I was unable to reproduce with lastpass and bitwarden and chrome on android.
I seems it may be related to bitwarden on your device specifically...


Fix source code of parts directory, to follow PEP8 rules

I have started building my own car.
I would like to contribute to this great project.
As a start point, I fixed source code from parts directory to follow PEP8 specification:
https://www.python.org/dev/peps/pep-0008/
In my opinion it is improving consistency of the source code.

Codecov Report

Merging #277 into dev will not change coverage.
The diff coverage is 84.61%.

@@           Coverage Diff           @@
##              dev     #277   +/-   ##
=======================================
  Coverage   54.58%   54.58%
=======================================
  Files          21       21
  Lines        1288     1288
  Branches      137      137
=======================================
  Hits          703      703
  Misses        561      561
  Partials       24       24

Impacted Files
Coverage Δ

donkeycar/parts/clock.py
0% <ø> (ø)
:arrow_up:

donkeycar/parts/transform.py
29.72% <ø> (ø)
:arrow_up:

donkeycar/parts/camera.py
30% <0%> (ø)
:arrow_up:

donkeycar/parts/keras.py
68% <100%> (ø)
:arrow_up:

donkeycar/parts/web_controller/web.py
52.43% <100%> (ø)
:arrow_up:

donkeycar/parts/simulation.py
94.44% <100%> (ø)
:arrow_up:

donkeycar/parts/actuator.py
38.23% <50%> (ø)
:arrow_up:

donkeycar/parts/datastore.py
77.41% <85.71%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7e843b...b253cef. Read the comment docs.

thanks


Handle Docker connection issues more gracefully

Fixes #376
Overview
This PR adds error handling that catches the case when Garden can't invoke Docker at all. The common cases of this we've seen are when the daemon is off and when the user isn't in the right group to use Docker so they get a PermissionDenied.
This PR also adds a new garden-ai docker check command. This is a sanity check command that users can run to confirm they have Docker set up correctly. They can also run garden-ai docker check --verbose to get the big old stack trace if that helps.
Discussion
Here's what it looks like.
Trying to run a notebook command but can't connect to Docker.

Running docker check when you can't connect.

Running docker check when you can connect :)

Testing
I tested the scenarios on my MacOS machine and on a Linux machine to find the error messages that Docker propagates.  I also added new unit tests for the error handling utilities.
Documentation
None yet, but next in line for me is #380

📚 Documentation preview 📚: https://garden-ai--389.org.readthedocs.build/en/389/

Codecov Report
Attention: 52 lines in your changes are missing coverage. Please review.

Comparison is base (7c4a36e) 75.69% compared to head (9bd2fe5) 76.04%.

Files
Patch %
Lines

garden_ai/app/notebook.py
26.92%
38 Missing :warning:

garden_ai/app/docker.py
35.00%
13 Missing :warning:

garden_ai/containers.py
96.66%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #389      +/-   ##
==========================================
+ Coverage   75.69%   76.04%   +0.34%
==========================================
  Files          42       44       +2
  Lines        2432     2542     +110
==========================================
+ Hits         1841     1933      +92
- Misses        591      609      +18

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Typescript definitions file

It would be nice to have a Typescript file for angular-clipboard in https://github.com/DefinitelyTyped/DefinitelyTyped/.
I can probably do this if you are not interested.

There's not much of an API and I don't personally use TypeScript, so feel free to submit something if you like 👍

@omichelsen I made a successful pull request into DefinitelyTyped for angular-clipboard.

Very nice, thanks!


Starting VSCode at Linux Ubuntu are too much slower

VSCode Version: 1.7.1
OS Version: #88-Ubuntu SMP Tue Dec 3 18:00:02 UTC 2013

Steps to Reproduce:

Start up VSCode spend up to 24' (minutes)

@ldeitos thanks so much for trying to run with --performance. It looks like all the time is spend on the main side of Code way before we even try to open a window. I wonder where that time is spend and unfortunately we do not have more detailed info in that area.
Do you see any output when you run code --verbose from the commandline?

@bpasero thanks to quick return! Unfortunately doesn't have any return at console stating with --verbose option.

Please see if you can find any logged output that could help us find the source of the issue:

run code --verbose from the command line and attach the output
in Code select Help | Toggle Developer Tools and select the Console
attach any output you see in the Console you can find

Thanks!

Hi @bpasero,
After a long wait...
OS console output:
[main 14:07:10] Starting VS Code in verbose mode
[main 14:07:10] from: /home/02844823904/Instaladores/VSCode-linux-ia32/resources/app
[main 14:07:10] args: { _: [],
  help: false,
  h: false,
  version: false,
  v: false,
  wait: false,
  w: false,
  diff: false,
  d: false,
  goto: false,
  g: false,
  'new-window': false,
  n: false,
  'reuse-window': false,
  r: false,
  performance: false,
  p: false,
  verbose: true,
  logExtensionHostCommunication: false,
  'disable-extensions': false,
  disableExtensions: false,
  'list-extensions': false,
  'show-versions': false,
  nolazy: false }
[main 14:07:13] IPC#vscode:startCrashReporter
[main 14:07:13] IPC#vscode:setHeaders
[main 14:07:14] IPC#vscode-workbenchLoaded

Apparently logs generated by --verbose option only starts to print a long time after send code start command.
VSCode console output:
Error: /usr/lib/i386-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /home/02844823904/Instaladores/VSCode-linux-ia32/resources/app/node_modules/native-keymap/build/Release/keymapping.node)(…)
Failed to clear temp storage: It was determined that certain files are unsafe for access within a Web application, or that too many calls are being made on file resources. SecurityError

@ldeitos is that output here:
[main 14:07:10] Starting VS Code in verbose mode
coming after 20 minutes only?

Yep

W.O.W

@ldeitos how about running just the Electron framework? this is the tool we use for all UI. You could get a version to double click from here: https://github.com/electron/electron/releases/tag/v1.3.7
Would be interesting to know if that opens also only after 20 minutes.

Ok, I will try, but only tomorow... Thanks!

Hi, I just download and run the Electron 1.3.7 standalone and that open instantaneously...
After that, I call ./electron ~/Instaladores/VSCode-linux-ia32/resources/app command to run VSCode and the result was the same: take 20+ min to open.
I try open another's simple applications through Electron and thats are fast to open... Any idea?

@joaomoreno any clue what could take 20 minutes until it reaches our main.ts#main() function?
I was first suspecting this to be caused by our CLI scripts, but it seems it also happens when you just run electron.
/cc @Tyriar for any clues?

@ldeitos so this is regular 32-bit Ubuntu?
This looks concerning:
Error: /usr/lib/i386-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /home/02844823904/Instaladores/VSCode-linux-ia32/resources/app/node_modules/native-keymap/build/Release/keymapping.node)(…)

We do require GLIBCXX 3.4.15+ https://code.visualstudio.com/Docs/supporting/requirements#_additional-linux-requirements

@Tyriar,
Unfortunately not... the Ubuntu is customized by operation area from company where I work... maybe that's the  problem.

I believe I may be seeing something related to this issue on Ubuntu 16.10.
I have never managed to wait long enough for /usr/share/code/code to start - it just appears to hang and do nothing, but my process list shows that a number of vscode related processes have kicked off:
jjw      20253 20251  0 23:52 pts/9    00:00:00 /usr/share/code/code --type=zygote --no-sandbox
jjw      20276 20251  0 23:52 pts/9    00:00:00 /usr/share/code/code --type=gpu-process --channel=20251.0.528142247 --mojo-application-channel-token=1AD72241F21E26AA8408450C08A0CCE2 --no-sandbox --window-depth=24 --x11-visual-id=33 --supports-dual-gpus=false --gpu-driver-bu rounds=5,18,56,70 --gpu-vendor-id=0x10de --gpu-device-id=0x1c20 --gpu-driver-vendor --gpu-driver-version --gpu-driver-date --gpu-secondary-vendor-ids=0x8086 --gpu-secondary-device-ids=0x191b --v8-natives-passed-by-fd --v8-snapshot-passed-by-fd

The weird thing is, if I start vscode using /usr/bin/code everything starts immediately and normally. I only discovered this by kicking off vscode from the command line rather than using the .desktop added to unity.

Hmm, I still haven't been able to repro this on 16.10. The difference between the 2 commands are:

via terminal: ELECTRON_RUN_AS_NODE=1 /usr/share/code/code /usr/share/code/resources/app/out/cli.js
via .desktop /usr/share/code/code

@bpasero @joaomoreno do you expect there to be a slowdown when just calling the electron binary directly without ELECTRON_RUN_AS_NODE=1?

I'm happy to help with tracking down this issue if I can, but running /usr/share/code/code --verbose --performance just produces the following 2 lines of output and hangs for >30mins:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

Let me know if there's anything else I can do...

Setting the env var ELECTRON_RUN_AS_NODE=1 and running /usr/share/code/code takes me immediately to a prompt.

I will try as soon as possible... I'm at vacation just now... on after 15/01 I will return and test the solution.

My startup issues seem to have been resolved with the 1.9.x release.

@joel-wright, sorry for the delay in responding!
Set env var as you recommended worked well, thanks for your attention.
As well, really after version 1.9 bug was fixed without set the variable.


Java Sorting Algorithm

Implement a sorting algorithm in Java, such as bubble sort, selection sort, or merge sort, and provide test cases to verify its correctness.

Can I work on this issue ?

Can I work on this issue ?

Can you assign me to this issue?
Thank you!

Sure thing @Janani-Balasooriya you can go ahead.


Checks if the page is read only before it alerts for navigating away

 JS pop up asking that there are unsaved changes for admins/assessors in readonly mode

Looks good! :+1:
Waiting for travis


Reading a file written with uproot: TTreeCache::{AddBranch,DropBranch}: unknown branch -> *

I'm taking a first stab at TTree writing and I'm running into an issue when trying to read the file back into ROOT (reading it back in uproot is working fine). ROOT is telling me
Error in <TTreeCache::AddBranch>: unknown branch -> *
Error in <TTreeCache::DropBranch>: unknown branch -> *

when I try to call TTree::Show on it. It seems like some kind of nameless ghost of a branch is appearing in the tree some how. I've also tested reading the uproot-originating tree via RDataFrame (to create ROOT histograms with RDataFrame::Histo1D). Things just crash.
My file is created like so:
## `out_dict` is a dictionary of {name(type=str) -> data(type=numpy.ndarray)}.
## `tree` is just a string for the tree name in the file.

branches = {}
for colname, arr in out_dict.items():
    if arr.dtype == np.uint32: ## convert to int32 for now
        out_dict[colname] = out_dict[colname].astype(np.int32)
    branches[colname] = uproot.newbranch(out_dict[colname].dtype)
    log.info(f"Saving branch {colname} with dtype {out_dict[colname].dtype}")

with uproot.recreate(outfile) as f:
    f[tree] = uproot.newtree(branches, flushsize="10 MB")
    f[tree].extend(out_dict, flush=False)

I've tried tinkering with the flush (not declaring a flushsize on the tree and giving extend flush=True) and compression settings just to see if it might impact the error I'm seeing but it doesn't (just file size is different, as I guess is expected). Link to file

Looks like we can get a branch named * through logic on the ROOT side, but I can't quite follow how this is triggered

Thanks for reporting the issue!

## out_dict is a dictionary of {name(type=str) -> data(type=numpy.ndarray)}.

To aid in debugging, could you tell me how many dimensions does data have? (data.ndim)

Preferably a full reproducer—we're trying to catch cases we implicitly don't handle and replace them with NotImplementedError messages for now, so that we understand the boundaries of the current implementation and at least prevent the impression that a correct file has been made.

(Multidimensional arrays are one such case.)
In your example, it looks like a branch with an illegal name is getting created. We don't know the full set of illegal names, but if we did, we should check for them. * is very likely not allowed because it can be used in SetBranchStatus to mean "all branches." I wonder if any wildcard character anywhere is not allowed?

Error in <TTreeCache::AddBranch>: unknown branch -> *
Error in <TTreeCache::DropBranch>: unknown branch -> *

I also get this when I open files with the ROOT Browser to explore files created with uproot.

I was able to reproduce the error.
@jpivarski This isn't related to multidimensional arrays like we thought. It is some error that crept in via some commit, that we didn't catch because surprisingly calling AsMatrix() on the tree works.

To aid in debugging, could you tell me how many dimensions does data have? (data.ndim)

All of the values in the dictionary are one dimensional arrays

Minimal reproducer below. As @marinang said, the error messages pop up when looking at a TBrowser (we can still see the data). Using TTree::Show only gives the error message; and trying to use RDataFrame just crashes.
import uproot
import numpy as np

data = {"a": np.random.randn(100).astype(np.int32),
        "b": np.random.randn(100).astype(np.float32),
        "c": np.random.randn(100).astype(np.float64)}

branch_templates = {"a": uproot.newbranch(np.int32),
                    "b": uproot.newbranch(np.float32),
                    "c": uproot.newbranch(np.float64)}

with uproot.recreate("output.root") as f:
    f["output_tree"] = uproot.newtree(branch_templates, flushsize="10 MB")
    f["output_tree"].extend(data, flush=False)

If the name of the branch literally is "*", then it should be visible as such in uproot. Uproot treats the branch names as simple bytestrings with no interpretation. When selecting branches in one of the methods like TTree.arrays, it uses glob patterns to match them, but you should be able to just say TTree.keys() to see a list of them all. Maybe the "*" branch is just the first where it starts going bad, due to some off-by-one error.
It is weird that TTree::AsMatrix didn't care, though. I would have thought that that called underlying TTree::GetEntry code...

@jpivarski The issue is at a more fundamental level.

If the name of the branch literally is "*"

No branch of that name exists. (Verified using uproot's .allkeys() and ROOT's TTree.Print())
uproot is able to read the data in all cases. ROOT's AsMatrix() works, but RDataFrame.AsNumpy() and TTree.Show() does not give the expected results (error).
I know this used to work before as I think I remember using RDataFrame to read data from files written by uproot - which is why I said the bad commit must have crept in at some point, will have to find it.
Since uproot is able to read it, the error is due to a lapse in our understanding of how TTree basket data is serialized. I have a rising suspicion that this may have something to do with the fBaskets attribute(the recovery basket data).. in which case this is not a trivial bug to solve.

@reikdas Do we know under what conditions it happens? Does this happen for all TTrees written by uproot (i.e. by only testing TTree::GetEntry and TTree::AsMatrix, we missed something that every TTree should have, which is only revealed when calling TTree::Show or RDataFrame::AsDict)? Does it happen for some or all of the unit tests

Does this happen for all TTrees written by uproot?

Yes :(

Does it happen for some or all of our unit tests?

All of them :(

Okay, so we have a mistake in our understanding of the TTree format because our readback tests in ROOT (TTree::GetEntry, which we used for what we thought was complete generality and TTree::AsMatrix, which we used for convenience) were insufficiently general. These two ROOT functions don't fully validate the binary format of the TTree object—they don't touch whatever we're doing that's wrong.
@douglasdavis and @marinang, what ROOT commands would constitute a sufficient test of conformance? I would think that RDataFrame is necessary (and I thought we used it in some earlier tests because it has a convenient RDataFrame::AsNumpy method).

RDataFrame::AsNumpy
TBrowser is something we can use in a spot-check, but not in tests, because of the GUI.
TTree::Show
TTree::Print

TTree::Show and TTree::Print can't be used in unit tests without capturing the stdout (and hoping ROOT doesn't change its human-readable format), but they can be included in the tests just to see that they don't crash.
Given that TTree::GetEntry and TTree::AsMatrix return expected results, the binary format that we're writing is probably not far from correct—only some functions touch the error; others don't.

TTree::Print

This works correctly on files written by uproot, so checking if this causes a crash is not a valid test.

what ROOT commands would constitute a sufficient test of conformance?

The items you listed exhaust all but one thing I can think of. Perhaps something along these lines can be tested (given some well understood ROOT file with pt and eta for example):
import ROOT
RDataFrame = ROOT.ROOT.RDataFrame
TH1DModel = ROOT.ROOT.RDF.TH1DModel
import uproot
import numpy as np

rdf = RDataFrame("tree", "file.root")
filter = rdf.Filter("pt > 25 && abs(eta) < 2.5")
pthist_model = TH1DModel("pthist", ";pt;", 25, 225, 20)
pthist = filter.Histo1D(pthist_model, "pt")

t = uproot.open("file.root").get("tree")
pt = t.array("pt")
eta = t.array("eta")
selection = (pt > 25) & (np.abs(eta) < 2.5)
counts, edges = np.histogram(pt[selection], bins=20, range=(25, 225))

pthist_data = np.ndarray(shape=(25,), dtype=np.foat64, buffer=pthist.GetArray())
assert np.allclose(pthist_data, counts)

@reikdas I'm triaging the issue and I came across this peculiarity: uproot does not produce TTree::fLeaves as it should. Reproducing example:
import numpy
import uproot
with uproot.recreate("byuproot.root", compression=None) as f:
    f["tree"] = uproot.newtree({
                    "branch": uproot.newbranch(numpy.int32, title="BRANCH")
                }, title="TREE")
    f["tree"]["branch"].newbasket(numpy.arange(10))

#include "TFile.h"
#include "TTree.h"

void byroot() {
  TFile* f = new TFile("byroot.root", "RECREATE");
  f->SetCompressionLevel(0);

  TTree* t = new TTree("tree", "TREE");
  int data;
  t->Branch("branch", &data, "leaf/I");

  for (int i = 0;  i < 10;  i++) {
    data = i;
    t->Fill();
  }
  t->Write();
  f->Close();
}

Reading back:
import uproot
byroot = uproot.open("byroot.root")
byuproot = uproot.open("byuproot.root")

byroot["tree"]["branch"]._fLeaves
# returns [<TLeafI b'leaf' at 0x7f0b43e44dd8>]
byuproot["tree"]["branch"]._fLeaves
# returns [<TLeafI b'branch' at 0x7f0b43e44ef0>]

# but
byroot["tree"]._fLeaves
# returns [<TLeafI b'leaf' at 0x7f0b43e44f98>]
byuproot["tree"]._fLeaves
# returns [None]

so they can't be identical binaries. This points to an issue in the writeobjany, our least favorite function, but at least we don't have to speculate that it's somehow outside of the TTree binary. Could you look into that while I dig deeper?

(Incidentally, uproot never uses TTree::fLeaves, so this would be a difference in usage that makes the files fully readable in uproot. It's a hint, at least.)

Another strangeness: I can read both files in ROOT!
% root -l byroot.root byuproot.root
root [0]
Attaching file byroot.root as _file0...
(TFile *) 0x55d38e4ba500
Attaching file byuproot.root as _file1...
(TFile *) 0x55d38e4aff80

and then
TTree *byroot, *byuproot;
_file0->GetObject("tree", byroot);
_file1->GetObject("tree", byuproot);
byroot->Print();

******************************************************************************
*Tree    :tree      : TREE                                                   *
*Entries :       10 : Total =             933 bytes  File  Size =        941 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Br    0 :branch    : leaf/I                                                 *
*Entries :       10 : Total  Size=        596 bytes  File Size  =        113 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*

byroot->Scan();

************************
*    Row   * branch.br *
************************
*        0 *         0 *
*        1 *         1 *
*        2 *         2 *
*        3 *         3 *
*        4 *         4 *
*        5 *         5 *
*        6 *         6 *
*        7 *         7 *
*        8 *         8 *
*        9 *         9 *
************************
(long long) 10

byuproot->Print();

******************************************************************************
*Tree    :tree      : TREE                                                   *
*Entries :       10 : Total =            1008 bytes  File  Size =       1820 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Br    0 :branch    : Int_t BRANCH                                           *
*Entries :       10 : Total  Size=        667 bytes  File Size  =        180 *
*Baskets :        2 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*

byuproot->Scan();

************
*    Row   *
************
*        0 *
*        1 *
*        2 *
*        3 *
*        4 *
*        5 *
*        6 *
*        7 *
*        8 *
*        9 *
************
(long long) 10

So far, so good. But I just remembered that "Scan" is different from "Show" (they both start with "S"), and I do see a difference when I "Show" them:
byroot->Show();

======> EVENT:9
 leaf            = 9

byuproot->Show();

======> EVENT:9

Specifically, the weirdness is in the list of leaves, which we already know is wrong from uproot. So far, this is the only thing I'm seeing as being wrong.

Thanks for pointing the thing about the fLeaves out! I'll look into it.

The issue with reading with RDataFrame is directly related to the malformed TTree::fLeaves list:
import ROOT
ROOT.RDataFrame("tree", "byroot.root").AsNumpy()
# returns {'branch.leaf': numpy.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)}

# but
ROOT.RDataFrame("tree", "byuproot.root").AsNumpy()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pivarski/miniconda3/lib/python3.7/site-packages/ROOT.py", line 418, in _RDataFrameAsNumpy
    column_type = df_rnode.GetColumnType(column)
Exception: string ROOT::RDF::RInterface<ROOT::Detail::RDF::RNodeBase,void>::GetColumnType(basic_string_view<char,char_traits<char> > column) =>
    Column "branch" is not in a dataset and is not a custom column been defined. (C++ exception of type runtime_error)

That error message

is emitted from RDFUtils.cxx's ColumnName2ColumnTypeName,
because colType fails to be filled by RDFUtils.cxx's GetBranchOrLeafTypeName,
which immediately calls TTree::GetLeaf (one string argument),
which calls TTree::GetLeafImpl (with nullptr as a branchname),
which wants to iterate over GetListOfLeaves (because branchname is nullptr),
which is simply fLeaves,
which uproot doesn't correctly write.

So unless there's something else wrong, fixing the TTree::fLeaves should allow uproot-written files to be read by RDataFrame.

I don't fully know how TTreeCache gets into the state reported by @douglasdavis and @marinang (though I have reproduced it with TBrowser), but that function

uses GetListOfLeaves at several points, so wrong values here can affect whether branch is initialized and nb is non-zero.
I think that a bname of "*" is a normal entry state for this function, but it presumably expects to replace "*" with a list of all branch names starting from the TLeaf object returned by GetListOfLeaves, which we already know is problematic.

So all warning bells are pointing at TTree::fLeaves. There's a good chance that fixing that one attribute will fix all the problems users have been seeing.
(At least let's hope!)

@reikdas Okay, here's the issue:

When TBranch::fLeaves is being written, it goes through writeobjany and writes whole TLeaf classes and objects (because it's first).
Then when TTree::fLeaves is being written, it goes through writeobjany and writes tags (references to the objects) that are too small by 8.

When reading (in _readobjany), I see a cursor.refs of
{278: <class 'uproot.rootio.TBranch'>, 469: <class 'uproot.rootio.TLeafI'>,
 465: <TLeafI b'branch' at 0x7fc57e7fc3c8>, 274: <TBranch b'branch' at 0x7fc57e835d68>}

when we want to read TTree::fLeaves. (This is after we've already successfully read all TBranch::fLeaves.) The leaf object should be referenced with tag 465, but the tag (first four bytes read by _readobjany) is 457, and it misses the TLeafI object by 8 (465 - 457). That's why it shows up as None in uproot and nullptr in ROOT: it has to be exact.
Interestingly, the correct position in the file I made with ROOT happens to be 457, but that could have been an accident of offsets. In both ROOT and uproot, changing the title of the TTree shifts all of these positions by a constant because they're all defined by relative seek positions. So it seems that uproot-writing is correctly not hard-coded for these positions, but it is off by 8. (I made files in ROOT and uproot with identical names apart from the leaf name, which is "leaf/I" in the file made by ROOT and "branch" in the file made by uproot, which happens to be the same number of characters.)
Some places to look for the reason behind the offset by 8:

The tags associated with classes have a relative seek position that differs from tags associated with objects by 4 (because tags associated with objects are relative to the seek position before reading the 4-byte tag and tags associated with classes are relative to the seek position after reading the 4-byte tag).
All of the tags in question have a kMapOffset of 2.
...?

Interestingly, the correct position in the file I made with ROOT happens to be 457

ROOT and uproot always write the same value of the tag, which is why the byte sequence in the TTree written by both ROOT and uproot are always the same. I had added in a -8 offset to make them match up (I thought it was relevant when debugging some unrelated issue).
I have fixed the issue in https://github.com/scikit-hep/uproot/commit/654713718651645dc0381a16c6c8da06cfc1d657 :)
Thanks so much for your help!

Is this in a PR? I'd like to see if this fixes everybody's problems.  :)

Yes https://github.com/scikit-hep/uproot/pull/358 ! (Sorry, I was cleaning some things up before making the PR - I am done now)

I've approved the PR and we should make a new version of uproot from it (your RDataFrame unit test shows that what you've done is already an improvement), but @douglasdavis and @marinang, does the new branch (or, by the time you read this, new version of uproot) fix your problems?

Testing the new branch, TTree::Show and TTree::Scan and RDataFrame seem to only find the first branch. (using the result from this comment. In a TBrowser everything is showing up.

I think I know what the problem is. I'll start looking into it. Thanks for reporting.

@douglasdavis @marinang Can you check if branch issue352 https://github.com/scikit-hep/uproot/tree/issue352 fixes your issue?

@reikdas I have just tried and I don't  see error message.

Error in <TTreeCache::AddBranch>: unknown branch -> *
Error in <TTreeCache::DropBranch>: unknown branch -> *

I also get this when I open files with the ROOT Browser to explore files created with uproot.

Things appear to be working smoothly for me as well.

Great! :)


[3.x] Improve event system

Summary
To allow for more flexibility we want to introduce improved events in Twill along with custom actions.
We need to:

[ ] Check current triggers of events within a save/edit/delete flow to reduce Laravel events to a single per action.
[ ] Implement a new custom actions system as illustrated below.

Check https://github.com/area17/twill/pull/1746


Depending on a release of a sub-project confuses the SBT plugin

Happens with the readme module of Ammonite: it transitively depends on com.lihaoyi:ammonite-ops_2.11, but does not depend on the sub-project that builds it. It only depends on a released version of it. But the InterProjectRepository of the SBT plugin is too naive, and confuses one for the other.
I have no idea of the amount of effort required to get a better view of inter-project dependencies from SBT.

Why does readme do that?

The readme sub-project depends on com.lihaoyi:ammonite-ops_2.11:0.4.8 via com.lihaoyi:scalatex-site_2.11:0.3.5. The coursier SBT plugin thinks this is a dependency towards the ops sub-project (that builds com.lihaoyi:ammonite-ops_2.11), thus puts nothing in the classpath for this particular module (SBT is supposed to put other sub-project classpaths in dependees itself). But SBT sees no dependsOn(ops) on readme, thus puts nothing in the classpath either. So ammonite-ops ends up missing in the classpath...

Fixed at least since 1.0.0-M14.


Add a way to show hedgehog

If we enter any query to config form we can’t go back to see hedgehog image.

I agree, too cute hedgehog. It's illegal to hide him.
I propose to show a placeholder with a hedgehog on an empty textarea.

We can add "× Clear" or "× Reset" button/link next to "Display compatible browsers".


docs: Add release process documentation

A small doc describing the formal release process.
Signed-off-by: Fabian Deutsch<EMAIL_ADDRESS>

@ifireball good point, added a section on the content. Please review.


Show the popup next to the word being typed

I just noticed the update to PowerToys and your work bundled in it, thanks a lot! I've missed this feature from macOS. However, I think that having the popup anywhere other than where the user is already looking is not optimal, especially on large monitors.
I'm not sure whether you could plug into that built-in autocomplete/autocorrect which comes with Win11, but it would be great to put the popup in a similar place

Thanks :)
It is a great feature, and it is currently experimental and you can activate it in PowerAccent (the option is not present in PowerToys).
But it is working only in some particular cases. I will need to find an another way to do it everywhere.

ref: https://github.com/microsoft/PowerToys/issues/20306


res.json

server/server.js
this is not necessary .serialize()
res.json(data)
will send this is json format for you.

I changed this to res.send and some text. Thanks!


Persistent identifiers (PIDs)

Our URIs, and especially those related to the digital objects, should be persistent. I suggest to assign Archival Resource Keys (ARKs) as they can be easily used in conjunction with the IIIF URI syntax:

Syntax
URI

ARK
[https://NMA/]ark:NAAN/Name[Qualifier]

IIIF Image Request
{scheme}://{server}{/prefix}/{identifier}/{region}/{size}/{rotation}/{quality}.{format}

IIIF Image Information
{scheme}://{server}{/prefix}/{identifier}/info.json

IIIF Resource (Manifest)
{scheme}://{host}/{prefix}/{identifier}/manifest

ARK+IIIF Example 1 a
https://iiif.participatory-archives.ch/ark:NAAN/<imageID>/full/max/0/default.jpg

ARK+IIIF Example 1 b
https://iiif.participatory-archives.ch/ark:NAAN/<imageID>/info.json

ARK+IIIF Example 2
https://iiif.participatory-archives.ch/ark:NAAN/<objectID>/manifest.json

Existing implementations that use IIIF in conjunction with ARK identifiers:

French National Library: https://api.bnf.fr/api-iiif-de-recuperation-des-images-de-gallica
Durham University Library: https://www.durhampriory.ac.uk/about-the-project/technology/ark-urls/

Cf. https://github.com/Participatory-Image-Archives/ark


[SPARK-24836][SQL] New option for Avro datasource - ignoreExtension

What changes were proposed in this pull request?
I propose to add new option for AVRO datasource which should control ignoring of files without .avro extension in read. The option has name ignoreExtension with default value true. If both options ignoreExtension and avro.mapred.ignore.inputs.without.extension are set, ignoreExtension overrides the former one. Here is an example of usage:
spark
  .read
  .option("ignoreExtension", false)
  .avro("path to avro files")

How was this patch tested?
I added a test which checks the option directly and a test for checking that new option overrides hadoop's config.

Test build #93196 has started for PR 21798 at commit 565e599.

Test build #93196 has finished for PR 21798 at commit 565e599.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93196/
Test PASSed.

Please, look at this PR: https://github.com/apache/spark/pull/21810 . It introduces AvroOptions.

Test build #93283 has started for PR 21798 at commit 3bd3475.

Test build #93283 has finished for PR 21798 at commit 3bd3475.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93283/
Test PASSed.

I added new option to AvroOptions and documented it. Please, look at the PR one more time.

LGTM except a comment

Test build #93363 has started for PR 21798 at commit 0657508.

Test build #93364 has started for PR 21798 at commit 3206a20.

Test build #93363 has finished for PR 21798 at commit 0657508.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroOptions(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93363/
Test PASSed.

Merged build finished. Test PASSed.

LGTM
Thanks! Merged to master.


Review and update Objects topic

From Velvia
Objects:
I wonder if its helpful or not to list, near the top, the number of bytes for each metadata field. That seems like a detail that most would not find interesting, but is also subject to change (although pretty rare).... just wondering.
Re shared objects.

moved to internal


📊 war: new dataset, metadata upgrade, new indicators

Summary

New dataset added "Countries in regions (COW): It counts the number of countries present in each region as defined by the State System Membership by COW.
Adds new indicators (those relating to "Conflict rate", more details in our upcoming article draft: Affects MIE, COW - Wars, and COW -MID
Update metadata propagation logic for all war datasets.
Improve metadata: make it less redundant by using Jinja, placeholders, etc.

Review
Some changes are related to the recent metadata upgrade, and others are more data-centric. I've assigned reviews of different files to different people.
@Marigold
I have been trying to leverage jinja templating, and placeholders in the metadata of some steps. I thought it'd be helpful to loop you in so you can see a real use-case scenario.
I'd suggest reviewing the following steps:

[ ] etl/steps/data/garden/war/2023-09-21/mie.*
[ ] etl/steps/data/garden/war/2023-09-21/cow_mid.*
[ ] etl/steps/data/garden/war/2023-09-21/cow.*

Some high-level (all you need to know) context on the changes in these files (they all 'suffered' from the same changes):

Add new indicators (new columns) derived from existing ones. This occurs with the call to the method add_indicators_conflict_rate, tb_regions, which also uses table tb_regions (new step dependency).
These new indicators preserve most of the original indicator's metadata (hence Jinja & placeholders are useful here!).

NOTE: step etl cow ーgrapher is failing when upserting to DB. I'm not quite sure what the error means:
ENV=.env.lucas etl cow --grapher
--- Detecting which steps need rebuilding...
OK (1.6s)
--- Running 1 steps:
--- 1. grapher://grapher/war/2023-09-21/cow...
2023-10-01 20:39:53 [info     ] upsert_dataset.verify_namespace namespace=war
2023-10-01 20:39:54 [info     ] upsert_dataset.upsert_dataset.start short_name=cow
2023-10-01 20:39:54 [info     ] upsert_dataset.upsert_dataset.end id=6202 short_name=cow
2023-10-01 20:40:05,957 - [bugsnag] WARNING - No API key configured, couldn't notify
Traceback (most recent call last):
  File "/home/lucas/repos/etl/.venv/bin/etl", line 6, in <module>
    sys.exit(main_cli())
   ...
  File "/home/lucas/repos/etl/.venv/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 614, in _only_one_row
    raise exc.MultipleResultsFound(
sqlalchemy.exc.MultipleResultsFound: Multiple rows were found when one or none was required

@paarriagadap
Added a new dataset to admin with number of countries in region per year, according to COW. Find preview in staging:

[x] etl/steps/data/garden/countries/2023-09-29/cow_ssm.*
[x] etl/steps/data/grapher/countries/2023-09-29/cow_ssm.py

@pabloarosado
Meta stuff:

Add warning if *.meta.yml file is missing.

[ ] etl/helpers.py

Add rename index function to Table.

[ ] lib/catalog/owid/catalog/tables.py

Support other excel formats in pr.read_excel

[ ] etl/snapshots.py

Updates to keep metadata propagation:
Some Garden step have been left out bc I assigned these to Mojmir.

COW - MID

[ ] etl/steps/data/meadow/war/2023-09-21/cow_mid.py

COW - Wars

[ ] etl/steps/data/meadow/war/2023-09-21/cow.py

MIE

[ ] etl/steps/data/meadow/war/2023-09-21/mie.py

Brecke

[ ] etl/steps/data/meadow/war/2023-09-21/brecke.py
[ ] etl/steps/data/garden/war/2023-09-21/brecke.py

PRIO

[ ] etl/steps/data/meadow/war/2023-09-21/prio_v31.py
[ ] etl/steps/data/garden/war/2023-09-21/prio_v31.py

UCDP

[ ] etl/steps/data/meadow/war/2023-09-21/ucdp.py
[ ] etl/steps/data/garden/war/2023-09-21/ucdp.py

Mars

[ ] etl/steps/data/meadow/war/2023-09-21/mars.py
[ ] etl/steps/data/meadow/war/2023-09-21/mars.py

NOTE: step etl cow ーgrapher is failing when upserting to DB. I'm not quite sure what the error means:

For some reason, we ended up with two identical origins in MySQL. This never happened before, I wonder what could have caused it. I couldn't replicate it locally. I pruned the table manually, and it works fine now. I'll look into how feasible would be constraints on origins table.


[+] Subprotocols support (Sec-WebSocket-Protocol header), fix #24

Subprotocol support fro server and client.
Server: conn.protocols now contain array of protocols requested by client
Client: options.protocols - array of protocols for requesting
Fix for #24

Thanks for the PR @VoidVolker
We'll have to add more to that API in order to follow the spec more closely. We have to:

[ ] check if the server answered a client handshake with one of the requested protocols
[ ] let the user provide a function to choose a protocol among those sent by the client

Yes, I know specs. I don't use this feature and don't know what interface for protocol select will be best. I fixed it just to get working. If I have time - I can try to add this features. Or, may be some one other can add this all.

I took a shot at implementing a more complete protocol negotiation: the server can list the protocols it considers valid or provide a custom implementation for the selection.
From the client perspective, I added a sanity check to the server response, as demanded by the spec.
What do you think?
I'm closing this PR to move forward with my implementation in master.
I'll add tests and release this when I got time to work on this again.


3 seconds execution delay

I am using it for a restaurant for printing on the thermal receipt printer.
However it has a delay around 4 to 5 seconds each time it gets executed.
Any ideas how to improve this ?
Maybe if we converted it to QT web service or QT windows service could be much faster?

Not sure why it takes so long for you, as it does not take that long for us. Maybe depends on the speed of the computer it is running on? Is it getting the HTML from the internet? Maybe it's slow getting the HTML?
If the issue is the time taken for the program to start running, then yeah, a web service would help. If it's the time to get the data over the internet, you would need to speed that up first.

Not sure why it takes so long for you, as it does not take that long for us. Maybe depends on the speed of the computer it is running on? Is it getting the HTML from the internet? Maybe it's slow getting the HTML?
If the issue is the time taken for the program to start running, then yeah, a web service would help. If it's the time to get the data over the internet, you would need to speed that up first.

I am using it with direct file:
PRINTHTML printername=cash1 file=receipt.html
But in my scenario I am printing 2 or 3 work orders in row.
This is the best solution I could find until now.
May be we could explore running the tool in a web service mode.
I would love to help on this.
Br
Yasir

I am using it with direct file:
PRINTHTML printername=cash1 file=receipt.html
But in my scenario I am printing 2 or 3 work orders in row.
This is the best solution I could find until now.
May be we could explore running the tool in a web service mode.
I would love to help on this.
Br
Yasir

This might be a good starting point for creating small web server found on this
link
#include <QtCore>
#include <QtHttpServer>

int main(int argc, char *argv[])
{
    QCoreApplication app(argc, argv);

    QHttpServer httpServer;
    httpServer.route("/", []() {
        return "Hello world";
    });

    httpServer.afterRequest([](QHttpServerResponse &&resp) {
        resp.setHeader("Server", "Super server!");
        return std::move(resp);
    });

    const auto port = httpServer.listen(QHostAddress::Any);
    if (!port) {
        qDebug() << QCoreApplication::translate(
                "QHttpServerExample", "Server failed to listen on a port.");
        return 0;
    }

    qDebug() << QCoreApplication::translate(
            "QHttpServerExample", "Running on http://<IP_ADDRESS>:%1/ (Press CTRL+C to quit)").arg(port);

    return app.exec();
}

I am thinking we can make a new executable like printhtmld (d for deamon mode), we can make it configurable to listen on specific PORT/ IP ADDRESS
We can define a routes like this

POST /printHtml
POST /printUrl
POST /printUrls

We can define a json object to have properties representing the args like this:
{
    "html": "
Hello World
",
    "printerName": "0",
    "title": "The world",
    "footer":  "0",
    "header": "",
    "topMargin": "0",
    "bottomMargin": "0",
    "rightMargin": "0",
    "leftMargin": "0",
    "secret" : "ASFaw4fa$FAdfasGA#$%garAGHSHTseh53ayEHsdfshfsdfSGFSdgear"
}

This might be a good starting point for creating small web server found on this
link
#include <QtCore>
#include <QtHttpServer>

int main(int argc, char *argv[])
{
    QCoreApplication app(argc, argv);

    QHttpServer httpServer;
    httpServer.route("/", []() {
        return "Hello world";
    });

    httpServer.afterRequest([](QHttpServerResponse &&resp) {
        resp.setHeader("Server", "Super server!");
        return std::move(resp);
    });

    const auto port = httpServer.listen(QHostAddress::Any);
    if (!port) {
        qDebug() << QCoreApplication::translate(
                "QHttpServerExample", "Server failed to listen on a port.");
        return 0;
    }

    qDebug() << QCoreApplication::translate(
            "QHttpServerExample", "Running on http://<IP_ADDRESS>:%1/ (Press CTRL+C to quit)").arg(port);

    return app.exec();
}

I am thinking we can make a new executable like printhtmld (d for deamon mode), we can make it configurable to listen on specific PORT/ IP ADDRESS
We can define a routes like this

POST /printHtml
POST /printUrl
POST /printUrls

We can define a json object to have properties representing the args like this:
{
    "html": "
Hello World
",
    "printerName": "0",
    "title": "The world",
    "footer":  "0",
    "header": "",
    "topMargin": "0",
    "bottomMargin": "0",
    "rightMargin": "0",
    "leftMargin": "0",
    "secret" : "ASFaw4fa$FAdfasGA#$%garAGHSHTseh53ayEHsdfshfsdfSGFSdgear"
}

Not sure it's necessary for a separate application, just add a -d parameter or something to have it sit there and listen for requests. I think it would be useful for a lot of folks.
I have not built this in a long time (I don't have the Qt stuff installed at the moment) but porting it to run with the latest Qt to pick up bug fixes in the HTML rendering engine would be a good idea also. I believe it can work with the latest Qt now (it was stuck on 4.8 due to QtWebKit being dropped).

Not sure it's necessary for a separate application, just add a -d parameter or something to have it sit there and listen for requests. I think it would be useful for a lot of folks.
I have not built this in a long time (I don't have the Qt stuff installed at the moment) but porting it to run with the latest Qt to pick up bug fixes in the HTML rendering engine would be a good idea also. I believe it can work with the latest Qt now (it was stuck on 4.8 due to QtWebKit being dropped).


Improve display of dependency changes when exceeding max length

We had a PR with no summary of changes, and I was wondering why we it didn't have one so I checked the code.
I've updated the logic to include a warning and as much of the changes as possible, when the change-list exceeds the max length.

Oh also this PR will truncate the trailing </details>, but I double-checked and things still work with that closing tag missing.


Make the search bar / sidebar less intrusive

Context
Currently we display our search bar to the left by default. This is awkward when the page has no sub-pages, because it is the only thing that exists in the left sidebar.
This is particularly strange on the landing page of sites, which almost never have sidebar content other than the search bar.
While it is possible to move the search bar to the header (navigation bar), most people will just go with the default.
At the same time, we have limited horizontal space in our navbar, so need to be careful not to take up too much space in general.
Ideas for discussion
Put the search box in the navbar, and grow it when active
We could make the searchbar default to the header. It would come to the left of most other header items.
In addition, we could "shrink" it by default, and expand it when it is active. For example, this is how the MDN documentation search bar works:

Use an icon to link to the search page
Another option is to only use an icon, rather than a form, if there is no sidebar content. For example:

The sidebar search form only shows up if there is some other sidebar content
Thus the sidebar is hidden entirely on any page with no sub-pages
By default, we put a search icon in the navbar that takes you to the search page.

Something like (apologies for the offset between cursor and icon, that's something weird with my screen):

Use some kind of UI "pop up" when a person clicks the search bar / icon
Finally, some sites use a search pop-up - for example, those that use Algolia have a search UX like this:

Proposal
We should agree upon an improved search UX so that we can make better use of the sidebar / navbar real-estate. See the proposed ideas above and lets try to converge on something.

I propose that we make the searchbar default to the header.

👍 on the proposal

It would come to the left of most other header items.

I agree with that idea.

In addition, I propose that we "shrink" it by default, and expand it when it is active.

I like that idea as well as long as it is on the left of all the other items... I do not like the behavior in the gif you posted above where the theme selector moves because of the search bar enlargement.

Another idea then would be to simply use an icon in the navbar header. Something like:

The sidebar search form only shows up if there is some other sidebar content
Thus the sidebar is hidden entirely on any page with no sub-pages
By default, we put a search icon in the navbar that takes you to the search page.

Something like:

Cc @jorisvandenbossche @12rambau in case they have ideas

I like the proposal as well!
here is a codepen with 3 options that I found relevant here. the only issue is that all these solution assume that the search bar will be in the navbar, which is not the case.

I'm a fan of the first example in that codepen!

I also like the first option, I think it is a +1 to keep the other links "findable", IMHO.

the only issue is that all these solution assume that the search bar will be in the navbar, which is not the case.

So, let's move it there 😉.

Another proposal: only show search if there is other sidebar content.
I just thought of a way that we could improve the UX with a pretty minimal change. Essentially, we could try to add logic in the search component so that it only shows up if there is other sidebar content. This means that on any page with sub-pages, it will show up. But on pages where it would only be the search and nothing else, it won't show up.
The main downside of this is that you wouldn't be able to get directly to the search button from the landing page of a site (assuming no other sidebar content). But maybe not such a big deal? I wanted to throw it out there since it seems pretty simple.

As long as this can easily be overridden I don't mind, but for the record, the majority of my interactions with our own docs and those of other projects using this theme involve going to the homepage, finding the search box, and searching for whatever function or method I've forgotten how to use. So I'd be sad if those other projects suddenly didn't have homepage search boxes

Yeah I have the same feeling...hmmm, I guess it would be best if we could get it into the header somehow without making it too crowded.

I guess I wonder if that's even necessary? Theme users can override the template for all pages or just specific pages... So they can move the search box wherever they want. Am I missing something?

I agree - but I think we still want the default behavior to be optimal somehow. It bugs me that the default landing page has a big empty sidebar because it has no sidebar links by default

ok, I see. no objection to having it in the topbar on every page as the default. I guess an alternative would be to make it so that there is, in fact, something in the sidebar on the landing page by default that isn't there on other pages. (like for MNE-Python we have "quick links" box)... not sure if there's something sufficiently generic that would fit that bill though?

I like the idea of an overlay (presumably load time is zero or negligible?). I don't like the idea of navigating to a separate URL to do the search. I also like it fine the way it is 🙂

See the following PR for one implementation of this.

https://github.com/pydata/pydata-sphinx-theme/pull/744

It takes the simplest approach (directing to the search page). We could try an overlay but this would be a lot more complex I think... Might be possible though

OK I realized this actually isn't too complicated given the current state of that PR, so I've added in a little search overlay there:
https://github.com/pydata/pydata-sphinx-theme/pull/744