Setting custom headers for 3rd party relays

Hello I am trying to set a custom header via the flashbots provider but can't seem to figure out where to set it. I see in the flashbots provider it takes a header in put here
    headers = self.get_request_headers() | {
        "X-Flashbots-Signature": f"{self.signature_account.address}:{signed_message.signature.hex()}"

but I cannot find where get_request_headers() is looking for the header data. If someone has an example using the flashbots wrapper with custom headers, please drop an example if you can!
Thanks

Turns out I didn't look hard enough. The header will be set when starting the HTTPProvider, not when initializing flashbot.
For anyone looking for this answer:
some_kwargs = {
'headers':
{"Authorization": "Token"},
"timeout": 30
}
web3fb = Web3(HTTPProvider(NODE_URL, request_kwargs=some_kwargs))

Turns out I didn't look hard enough. The header will be set when starting the HTTPProvider, not when initializing flashbot.
For anyone looking for this answer:
some_kwargs = {
'headers':
{"Authorization": "Token"},
"timeout": 30
}
web3fb = Web3(HTTPProvider(NODE_URL, request_kwargs=some_kwargs))


Return used and allocated values in combined_quota results even when no quota defined

We were not calculating nor displaying used and allocated values for quotas that were not defined. This enhancement enables that.
https://bugzilla.redhat.com/show_bug.cgi?id=1415792

Before

After


PRCI - Authentication

Developers must be able to configure whether our component will prompt the user for authentication before or after the request, for each demand type. See blindnet-io/product-management#814

@milstan @m4rk055 From our dicussion last week regarding authentication, specifically the a priori and a posteriori cases, I realize I may have misunderstood it originally when working on the sketches.
I currently plan for it to look something like this, depending on the two different auth cases. However, I didn't think the request was actually submitted in the a posteriori case until the user logged in - but as @milstan explained, the request should be submitted and then the authentication is requested.
So, I'm thinking perhaps the a posteriori case is better handled through the status view. I.e. a user submits their request, is brought to the status page which displays in processing, and then once the the request is processed the state switches to require_authentication, then the PRCI will show a status indicating this and the user can then authenticate within the PRCI via the methods configured. @m4rk055 Is this how you imagine it working?

I haven't given it much thought yet.
What you described works when authentication is immediately requested but gets tricky when auth request arrives asynchronously (could take days e.g.) - how is the data subject then notified?
When e.g. an unauthenticated ACCESS request is made, what exactly are the possible workflows to authenticate?
@milstan talked about one way to authenticate - by a third party service that requests and verifies users' identity documents.
If a data subject is authenticated outside of PRCI, it's weird to request them to open the request status page to get authenticated in the devkit.
In that case, it's probably better for the server to provide the data subject's identity to the PCE directly (new API endpoint).
This is quite a complex issue and requires more information.

for a posteriori authentication, I like the idea of status page showing - pending authentication. Let us say that the Client system is responsible, in this case, to promt the user to authenticate, and let the devkit know when the user has been authenticated. At that point the satus page changes, and processing can begin.
So the status page would only be responsable for whoing "pending authentication" status, and not necessairly promting the user to authenticate.
I agree it's a complex issue, but we can also consider this to be a good enough current solution.


Refactor Aggregates to Behave Like Functions

There is virtually no difference between an aggregate and a function. With the current implementation they have limitations - mainly that they couldn't be used in a future having expression.

Coverage decreased (-0.3%) to 99.746% when pulling 2e42ae301cb604c8c58ac6cbb408334b26b58e78 on jeffgbutler:aggregates-are-functions into e7721be053b36a3d4dd9596b82086b553b26bd75 on mybatis:master.


OWASP Nettacker API

Hello everyone,
OWASP Nettacker doesn't have any API at the moment so It's best to develop one! The purpose to design an API is that we can provide more user interfaces and controllers.
where are the events? all events that we'd show to users are using core.alert to print on the command line in one of these types: info, warn, error, write. also, every thread and events will save in the output file with JSON structure. output filename by default is like results/results_date_random_chars.html -> results_2018_01_11_08_05_56_mkjiyjxklc.html and it's stored in log_in_file variable in the framework (even if users change it with -o switch it's still in that variable).
example of a tcp_connect_port_scan event:
{
  "USERNAME": "",
  "CATEGORY": "scan",
  "HOST": "<IP_ADDRESS>",
  "DESCRIPTION": "OPEN PORT",
  "TIME": "2018-01-11 11:47:05",
  "PASSWORD": "",
  "TYPE": "tcp_connect_port_scan",
  "PORT": 1080
}

this JSON has 7 keys which are HOST, PORT, USERNAME, PASSWORD, DESCRIPTION, CATEGORY, TIME, TYPE. TYPE is the module name. I should notice that the keys are changing depend on the language, so if somone select Persian language, HOST will change to هاست and these replacements are available in core/languages.py.
{
  "درگاه": 1080,
  "هاست": "<IP_ADDRESS>",
  "کلمه عبور": "",
  "دسته": "scan",
  "نوع": "tcp_connect_port_scan",
  "زمان": "2018-01-11 11:47:36",
  "نام کاربری": "",
  "توضیحات": "درگاه باز"
}

replacements:
_HOST = messages(language, 53)
_USERNAME = messages(language, 54)
_PASSWORD = messages(language, 55)
_PORT = messages(language, 56)
_TYPE = messages(language, 57)
_DESCRIPTION = messages(language, 58)
_TIME = messages(language, 115)
_CATEGORY = messages(language, 116)

to check the replacements you can open the core/languages.py and look for the keys. _HOST key is53 so it's (default: "en"):
{
  "53": {
    "ru": "хозяин",
    "fr": "HÔTE",
    "en": "HOST",
    "nl": "HOST",
    "el": "ΠΛΗΘΟΣ",
    "vi": "HOST",
    "de": "GASTGEBER",
    "tr": "HOST",
    "ps": "HOST",
    "ur": "HOST",
    "fa": "هاست",
    "ja": "ホスト",
    "hi": "मेज़बान",
    "zh-cn": "主办",
    "ko": "숙주",
    "it": "OSPITE",
    "hy": "HOST",
    "ar": "مضيف",
    "id": "TUAN RUMAH",
    "es": "ANFITRIÓN"
  }
}

at the end, I should mention that the default language is always "en" except if a user changes it with command line or config file. the language value stores in language variable in the framework. if a value doesn't have any replacement in other languages, you may use en.
let me know if there is any question.
Regards.

Hello,
Can you let me know the features that are yet to be implemented in the API and if I could possibly work on some of them!
Regards.

Hello,
Actually i want the same thing for the APIs frontend yet to be implemented as mentioned above by @shaddygarg  so we can  work on them.
Regards.

Hi,
the API is completed. I gonna update documents regarding this, then we can discuss the new ideas.
Regards.


On BigTable read_row(), provide default to retrieve only most recent cell values

The current signature on the Bigtable Table read_row() method has a default of filter_=None:
def read_row(self, row_key, filter_=None):
    ...

In cases where a cell value may have been updated multiple times, the default will be to return the full time series with timestamps for each value which can slow down read performance in a non-obvious way.
In the current Python API the cells() method on row_data (PartialRowData) makes a deep copy of the cells, which compounds the performance issue.
@property
def cells(self):
    """Property returning all the cells accumulated on this partial row.

    :rtype: dict
    :returns: Dictionary of the :class:`Cell` objects accumulated. This
              dictionary has two-levels of keys (first for column families
              and second for column names/qualifiers within a family). For
              a given column, a list of :class:`Cell` objects is stored.
    """
    return copy.deepcopy(self._cells)

Consider:

Making a default filter on read_row() to retrieve only the most recent value of any cell unless the full or partial time series is requested.
Allowing a ColumnFamily to implicitly or explicitly limit cells to only one value (no timeseries).
Adding a cell_value(column_family_id, column_id) method to row_data (PartialRowData) to allow more efficient retrieval of a single cell value.

Thanks for filing @zakons!

I don't think it's a good idea to have any default filter. However, I think we could provide some kind of easy to access filter, i.e. as a constant in the package google.cloud.bigtable.MOST_RECENT_FILTER or maybe as a class constant in Table so you could write code like
row_data = table.read_row(row_key, filter_=table.MOST_RECENT_FILTER)

I'm not quite clear on your ColumnFamily suggestion. The feature set of the actual backend API is all we can really expose. Is there a particular method where you'd like to see different behavior? (Most API calls that actually refer to a column family just use a string ID for the column family, not an actual ColumnFamily instance.)

Adding cell_value() seems very easy to implement and not at all controversial to add to the API surface.

/cc @garye

I like the table.MOST_RECENT_FILTER approach.  We can also add in table.KEY_ONLY_FILTER, which is useful for just doing things like row counts.

I'm not certain about ColumnFamily either.  Is it the same as 1.?

+1

@dhermes, @garye is less work on the client-side these days.  I am working with other developers across all languages for Cloud Bigtable.

Thanks.

I believe the name table.MOST_RECENT_VALUES_FILTER or something similar connoting what it is that is recent would be more clear.
I agree with your point that we should not be suppressing any existing backend functionality in our API.

Appreciate the quick feedback.

Note that the cell_value(column_family_id, column, index=0) method returns an immutable value, so the underlying Cell's, which are mutable, are not exposed to the user.
Also, add a cell_values(column_family_id, column) iterator or generator method to row_data (PartialRowData) which will provide an iteration over all the cell values, returning index, timestamp, value - each of which are immutable.  The first index in the iteration would be 0, or most recent value.

@dhermes, can we please discuss design issues for PR #4564 here?

Sure. Go ahead and discuss? I am 100% for dropping the deepcopy, but I'd prefer to do it as an opt-in, so users can knowingly get mutate-able values.

@zakons: can this be closed?


'Pages' section of Report capped at 5?

Hello! Love the app but would really benefit from being able to specify how many pages I want to see in the report. The documentation says:

It also includes a drill down of the top five countries and pages (by total number of Web Vitals events received), so you can see if certain pages or user populations perform better or worse than others.

I know for a fact we're sending web vitals info from thousands of pages, so not sure why I can only see five? Also curious as to how it choses said pages. Is it based off performance? They seem to be a mix of good and bad performing pages.
Anyhow, any help would be much appreciated, thank you!

So, I understand this might not be the answer you were looking for, but part of the intention with releasing this tool was to provide developers with an example that they could fork and build on top of. It was never intended to be full-featured reporting tool.
Are you able to follow the instructions to build and run the tool yourself? If so then you could change this line of code from .slice(0, 5) to .slice(0, 20) (or whatever number you want).
I'll keep this issue open to track this feature request, but as I said our hope was that developers would build their own dashboards based on this example. My team unfortunately doesn't have the resources to actively maintain and develop this tool.

@philipwalton Thank you for thoughtful response! I see, that makes sense, I hadn't done that because I hoped I was just missing something obvious from the config. I'll give forking + running the tool myself a shot, cheers


[Reviewed] [Gamepad] Handle advanced vibrations

Update the Gamepads extension, from 0.4.1 to 0.5.0.
This add new features:

A new action: Advanced gamepad vibration which use the strong and weak magnitude.
A new action to edit the magnitude in real time.
An expression to get the strong magnitude.
An expression to get the weak magnitude.

Thanks to @lhphr & @ZachjuKamashi
The project example with the extension:
Extension.Example.zip

I also just noticed a typo for the condition that checks which button was pressed. "BAck" should be "Back" instead

Right, nice catch!
I made the changes you asked for.

I made the edit I hope it's what you talking about.
The extension is still functional.
I don't have two controllers so I couldn't test it.

Thanks!
@lhphr & @ZachjuKamashi if you have a feedback or not let me know, then I merge your new functionalities.

Alright I've tested it with two controllers and I discovered no issues. Both controller can be vibrated separately and changed separately. I discovered no issues with expressions either so I think it's good to go! 👍

Awesome, thanks everyone!
This is now merged, and is available in few minutes in the app.
Again thank you for these new functionalities and tests.
I taking care of updating examples with this new version.


release build for windows

I'm not sure what's the reason for not having windows build. i searched through the issues and found nothing.
I want to to test kubefed by creating clusters with kind and join them with kubefedctl but kubefedctl is not available for windows.

#1101 related.  Release on Chocolatey or other Windows package manager could be helpful too.
https://chocolatey.org/docs/create-packages
git clone --depth 1 https://github.com/kubernetes-sigs/kubefed.git
cd kubefed\cmd\kubefedctl
go build -o kubefedctl.exe main.go

/close


Updated refreshPropertyPane => context.propertyPane.refresh();

Updated this.refreshPropertyPane() to this.context.propertyPane.refresh(); as is the case with RC0 onwards as the documentation was misleading on this.

Q
A

content fix?
yes

New article?
no

Related issues?
N/A

What's in this Pull Request?
Updated this.refreshPropertyPane() to this.context.propertyPane.refresh(); as is the case with RC0 onwards as the documentation was misleading on this.

@KoenZomers,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

thx Koen,
I'll perform also some additional updates to ensure that this is fully updated to GA level - and remove the marker from the start of the page. This is good start for that. thx.

You're welcome @VesaJuvonen . I went through this one with a customer. The this.refresh()  also doesn't seem necessary anymore in GA. Do you want me to submit a pull request to have that removed as well?


Fixes for named pipe dialer

1 - add a pipe parameter so users can encode it in a URL DSN
2 - avoid unneeded browser queries when TCP protocol is first in the dialer list

I just noticed that continuous-integration/appveyor/pr  check is failing.
Can you look into that? I couldn't restart it to see if it's a transient issue.

That test is flaky, it passes most of the time. I need to figure out if there's a real driver bug causing it but I haven't made time to do it.


Native macOS menu last duplicate

On macOS, when adding the MainMenu (natively) I get the last menu entry duplicated:

OK, I figured out that it expects Window menu, and I had just the Application menu present.
Is there a quick way to position (my custom, ImGui one) Help menu after the Window menu?

It's been a while since I worked on the native Mac menu code.  I'll take a peek this weekend and see if anything comes back to me, although honestly, I can't remember actually trying to add a Help menu (nor a Windows menu).  There may be some special requirements for these.

I managed to get to a point where I have all the menu items working fine - the only last thing is the ordering. The last menu item is "Window", and Help goes before it (Window menu is created in the resources, Help is created from the code using the OsBeginMenu API). I'd like to have Help menu at the end (as it is in every app on macOS). Tried to play with the logic in the code but there was always some unexpected behaviour with the indexing and I didn't get it right. If you could have a quick look I'd appreciate it!

The indexing behavior was indeed tricky.  It took me a while to figure out how to do this.  I decided to add a bool flag to the menu to indicate that you're creating a Help menu.  Another option was to try to automatically detect it via the menu name.  Apple seems to do this, but that felt kind of gross to me, but they're probably more robust, doing it for all support languages or something.  I didn't see a Cocoa API function that indicated that this was a help menu, which would solve the problem nicely, so I went with the flag.  Still another option would be to create a BeginHelpMenu() function.  Feedback on these options is welcome.
It's actually required to create that menuitem last, because it's doing some trickery with the index to make sure it goes last, past the default Window menu.
I've checked in a patch on the development branch.  The Features example now demonstrates a help menu with an About popup.  Take a look and see if this works for you.  If so, I'll merge it to the main branch.

Thanks for looking into it!
I just tried it and there seems to be some issues still.
First of all the assert in OsBeinMenu fails, because Help is the last menu, but has a submenu in it (hence another call to OsBeginMenu before OsEndMenuBar is called). But even when I ignore the assert my Window menu is gone and I end up with duplicated Help menu at the end. I got the same when tried to modify it myself earlier.
Maybe I will have a little more time tomorrow to look at it again.
Thanks anyway for taking the time!

Ah, yeah... I guess this approach only works with a single additional menu after the Window menu.  I'll see if I can figure out how to make it work with any number of submenus.   May need to re-think the approach a bit.

Give the latest development version a try.  I've changed the API to add ImFrame::BeginHelpMenu() instead of a flag to mark the first entry after the Window menu, which I think is a little cleaner.  The Features example now tests dynamically adding menus before and after the Window menu, and it seems reasonably robust.

Thank you - the version works well for me! I had to fix one small detail to prevent a crash, but otherwise it works as expected. Appreciate your time!


Using Ansible to create Docker "Layers" in DrupalVM

Hi Guys,
Firstly, great work. I am using drupal-vm docker image and its working fantastic.
I was wondering if it would be a good approach to create layers on top of the base drupal-vm using Ansible?
Eg:

docker-debian8-ansible (geerlingguy/docker-debian9-ansible:latest)
-- drupalvm (geerlingguy/drupal-vm)
---- drupalvm-extras (install just the specific Ansible roles that user needs)

I know in an ideal world monolithic containers are frowned upon, but seeing as we are using this approach with Drupal VM, would this be a good way to "Layer"?

Hi,
I did a quick test and all seems to work fine. What I have is something like:

docker-debian8-ansible (geerlingguy/docker-debian9-ansible:latest)
-- drupalvm (dakku/drupal-vm) (fork & modified slightly to remove extras and make it a base image)
---- drupalvm-extras (custom Dockerfile with custom config.yml - basically layer on top all extras I need)

This is the content of my drupalvm-extras Dockerfile
FROM dakku/drupal-vm:base
LABEL maintainer="Dakku Singh"

# Copy provisioning directory, variable overrides, and scripts into container.
COPY ./docker-hub-overrides.yml /etc/ansible/drupal-vm/local.config.yml

# Provision Drupal VM inside Docker.
RUN ansible-playbook /etc/ansible/drupal-vm/provisioning/playbook.yml

and this is the content of my docker-hub-overrides.yml
---
hostname_configure: false
firewall_enabled: false
installed_extras:
  - drush
  - mailhog
  - pimpmylog
  - drupalconsole

# Don't build or install Drupal inside the container.
drupal_build_makefile: false
drupal_build_composer: false
drupal_build_composer_project: false
drupal_install_site: false

Incase anyone is interested in the "Layered" approach I have taken, you can check out:
https://github.com/dakkusingh/drupal-vm
https://github.com/dakkusingh/drupal-vm-extra
Cheers again for the fantastic work.


Create open_redirect_emp-eduyield.yml

Description
Add new coverage for a redirect which chains google amp to evade destination url limits.
Associated samples
Associated hunts
Samples matching L365

Hunt 1

/update-test-rules

hits look solid in shared eml, pretty tight rule eduyield.com and emp.eduyield.com have no MX records.


Build TensorFlow Lite for ARM64 boards failed

System information

Linux Ubuntu 16.04
TensorFlow installed from (source or binary): source
TensorFlow version: r1.14

Describe the problem
I got build error: undefined reference to `NnApiImplementation()'
Provide the exact sequence of commands / steps that you executed before running into the problem
I followed https://www.tensorflow.org/lite/guide/build_arm64 . Here are the exact sequence of commands:
git clone https://github.com/tensorflow/tensorflow && cd tensorflow
git checkout -b r1.14 origin/r1.14
./tensorflow/lite/tools/make/download_dependencies.sh ./tensorflow/lite/tools/make/build_aarch64_lib.sh
Any other info / logs
I got these error info when I execute build_aarch64_lib.sh :
/home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/lib/libtensorflow-lite.a(nnapi_delegate.o): In function tflite::NNAPIAllocation::~NNAPIAllocation()': nnapi_delegate.cc:(.text+0x28): undefined reference to NnApiImplementation()'
/home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/lib/libtensorflow-lite.a(nnapi_delegate.o): In function tflite::NNAPIAllocation::NNAPIAllocation(char const*, tflite::ErrorReporter*)': nnapi_delegate.cc:(.text+0x1a4): undefined reference to NnApiImplementation()'
/home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/lib/libtensorflow-lite.a(nnapi_delegate.o): In function tflite::NNAPIDelegate::~NNAPIDelegate()': nnapi_delegate.cc:(.text+0x218): undefined reference to NnApiImplementation()'
nnapi_delegate.cc:(.text+0x234): undefined reference to NnApiImplementation()' /home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/lib/libtensorflow-lite.a(nnapi_delegate.o): In function tflite::addTensorOperands(tflite::Subgraph*, ANeuralNetworksModel*, unsigned int*, std::vector<long, std::allocator >*)':
nnapi_delegate.cc:(.text+0x2b8): undefined reference to NnApiImplementation()' /home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/lib/libtensorflow-lite.a(nnapi_delegate.o):nnapi_delegate.cc:(.text+0x578): more undefined references to NnApiImplementation()' follow
collect2: error: ld returned 1 exit status
tensorflow/lite/tools/make/Makefile:227: recipe for target '/home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/bin/minimal' failed
make: *** [/home/administrator/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/bin/minimal] Error 1
make: *** Waiting for unfinished jobs....

This should be fixed on head. Note that the 1.14 branch is in a state of flux and will be updated with the fix. Please re-open if you find otherwise (testing against head).


Fix Teams workflow webhook

⚠⚠⚠ Since we do not accept all types of pull requests and do not want to waste your time. Please be sure that you have read pull request rules:
https://github.com/louislam/uptime-kuma/blob/master/CONTRIBUTING.md#can-i-create-a-pull-request-for-uptime-kuma
Tick the checkbox if you understand [x]:

[x] I have read and understand the pull request rules.

Description
This commit backports a slightly modified version the Adaptive-Templeate found in the Master-branch.
The version of the template used is from commit 2c31f3a2ff37d96331b0dc14d896eea37f76c47a.
All previously created connector webhook will keep working until Microsoft removes this functionality October 1st 2024.
https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/
All new webhooks set up via workflows will use the backported version.
Line 13 in Teams.vue might need to be modified later. Teams.vue
I was not able to find any Microsoft documentation that can replace that link that this point in time.
Fixes #4934
Fixes #4922
Type of change

Bug fix (non-breaking change which fixes an issue)
This change requires a documentation update

Checklist

[x] My code follows the style guidelines of this project
[x] I ran ESLint and other linters for modified files
[x] I have performed a self-review of my own code and tested it
[ ] I have commented my code, particularly in hard-to-understand areas (including JSDoc for methods)
[x] My changes generates no new warnings
[ ] My code needed automated testing. I have added them (this is optional task)

Screenshots (if any)
Connector webhook (legacy)

Workflow webhook

Hi @Tabisch In case you don't have time to follow up, I created a PR for your branch on @CommanderStorm 's requirement.
https://github.com/Tabisch/uptime-kuma/pull/1
Please feel free to have a look.

@tpai
Looks good to me.
Merged it.
Totally forgot that this was still open.
Thank you for cleaning that up for me.

@CommanderStorm Can you please review this PR again 🤗

@CommanderStorm ping :worried:

I don't think investing time into this PR is worth it anymore.
At least, I don't have the spare time right now..
This is resolved in https://github.com/louislam/uptime-kuma/releases/tag/2.0.0-beta.0
=> please test that release instead


Hubzilla addons undetected in a fresh installation

Currently, my instance is not detecting the bundled addons (as it can be seen here). On the settings/featured section, where the addons are controlled, I'm greeted by a message that says "No feature settings configured". Why are the addons disabled? Is there some configuration required to enable them by default?

Did you follow the installation instructions for the addons in the installation documentation? https://github.com/redmatrix/hubzilla/blob/master/install/INSTALL.txt#L131

Once they have been physically installed on the server, the hub adminsitrator needs to enable any which are desired for this site at admin/plugins . Then the addon settings and configuration for those plugins are available to site members.

@jeroenpraat: Been there, done that. Though I still can't figure out where is the admin site (it's not available in the installation folder nor at https://social.azkware.net/admin ).

Are you sure your account is an admin account? If not follow this: https://github.com/redmatrix/hubzilla/blob/master/install/INSTALL.txt#L191

For some reason my user was not set as the admin by default, I had to manually set the permissions in the database. But after that, the admin page was finally open for business. Thanks for your help!

You're welcome. Good luck with your Hubzilla hub,


Add xMatters as an Alertmanager Webhook Receiver

This allows users to send alerts into xMatters.

Thanks!
Please note that the URL is not considered secret and is displayed in the web UI. You could use the http_config token to get API keys, instead of using the URL.

Is there a way in http_config to set a parameter, or would we have to go with bearer token or basic auth instead?

You would need basic auth ir bearer token. Note that I am pretty sure that the username of basic auth is not mandatory.

I'll add some details on API Key security for our users who want that. Is there anything else I need to do?

@roidelapluie I've added a note telling our users the lack of security with the way the API Key is stored and mentioned the option of basic authentication. Is there anything else I need to do for our integration to be in the docs?

Thanks! It is unclear how to use basic auth however, maybe you should consider adding an exemple.


[youtube] Format 22 has been removed by Youtube

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that yt-dlp is broken on a supported site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
Europe
Provide a description that is worded well enough to be understood
When attempting to download a video from Youtube in the listed format "22" (1280x720 MP4 video), multiple manual reinvocations of yt-dlp are required to download the video. Apparently out of a sudden, the correct stream will appear.
I understand that this ticket is not fully compliant with the requirement to post verbose output. However, this appears not mainly a yt-dlp error but rather some tampering by Google / Youtube with the stream info, possibly to sabotage downloader programs like yt-dlp.
I have redacted the stream info for privacy reasons.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
$ yt-dlp --proxy="" -f22 "https://www.youtube.com/watch?v=xxxxxxxxxxxxx"
[youtube] Extracting URL: https://www.youtube.com/watch?v=xxxxxxxxxxxxx
[youtube] xxxxxxxxxxxxx: Downloading webpage
[youtube] xxxxxxxxxxxxx: Downloading ios player API JSON
[youtube] xxxxxxxxxxxxx: Downloading m3u8 information
ERROR: [youtube] xxxxxxxxxxxxx: Requested format is not available. Use --list-formats for a list of available formats
$ yt-dlp --proxy="" -f22 "https://www.youtube.com/watch?v=xxxxxxxxxxxxx"
[youtube] Extracting URL: https://www.youtube.com/watch?v=xxxxxxxxxxxxx
[youtube] xxxxxxxxxxxxx: Downloading webpage
[youtube] xxxxxxxxxxxxx: Downloading ios player API JSON
[youtube] xxxxxxxxxxxxx: Downloading m3u8 information
ERROR: [youtube] xxxxxxxxxxxxx: Requested format is not available. Use --list-formats for a list of available formats
$ yt-dlp --proxy="" -f22 "https://www.youtube.com/watch?v=xxxxxxxxxxxxx"
[youtube] Extracting URL: https://www.youtube.com/watch?v=xxxxxxxxxxxxx
[youtube] xxxxxxxxxxxxx: Downloading webpage
[youtube] xxxxxxxxxxxxx: Downloading ios player API JSON
[youtube] xxxxxxxxxxxxx: Downloading m3u8 information
ERROR: [youtube] xxxxxxxxxxxxx: Requested format is not available. Use --list-formats for a list of available formats
$ yt-dlp --proxy="" -f22 "https://www.youtube.com/watch?v=xxxxxxxxxxxxx"
[youtube] Extracting URL: https://www.youtube.com/watch?v=xxxxxxxxxxxxx
[youtube] xxxxxxxxxxxxx: Downloading webpage
[youtube] xxxxxxxxxxxxx: Downloading ios player API JSON
[youtube] xxxxxxxxxxxxx: Downloading m3u8 information
[info] xxxxxxxxxxxxx: Downloading 1 format(s): 22
[download] Destination: xxxxxxxxxxxxxxxxxxxxxxx [xxxxxxxxxxxxx].mp4
[download] 100% of   xxx.xx MiB in 00:xx:xx at xx.xxMiB/s

This is mainly a side effect of the android client being removed, due to server-side attestation being added by google. Format 22 is no longer reliably available.
You will likely have to switch to 136+140 instead for reliable extraction. Do note that this requires ffmpeg for merging.

you may pass --impersonate safari to yt-dlp to get the
format 95 (1280x720 video+audio m3u8) or
96 (1920x1080 video+audio m3u8)

you may pass --impersonate safari to yt-dlp to get the format 95 (1280x720 video+audio m3u8) or 96 (1920x1080 video+audio m3u8)

can you please show a full command"
i get safari is not in the impersonate error???

--extractor-arg "youtube:player_client=mediaconnect" should also get you format 95 if you don't have curl_cffi installed

--extractor-arg "youtube:player_client=mediaconnect" should also get you format 95 if you don't have curl_cffi installed

my dude came in clutch.
i have installed that damn package on 5 version of python that i have its like it doesnt exist on my os.
anyway, your method worked. bit slow but fast with aria2 external downloader it was fast.
now time for me to modify all my scripts.
thank you

As of today, it seems that format 22 is gone for good: not available for this recent video, nor these previous videos where it was available before (NsGQSUdYjgE and P_so2nUob1Y).

[MacOS]
Adding --extractor-arg "youtube:player_client=mediaconnect" to my script does provide the 95 format option, but when I run yt-dlp --list-impersonate-targets, I get the following response even though I've installed brew install cffi.
[info] Available impersonate targets
Client   OS   Source
---------------------------------------
Chrome   -    curl_cffi (not available)
Edge     -    curl_cffi (not available)
Safari   -    curl_cffi (not available)

It's a bit of a shame that format 95 downloads very slowly, and does not provide a preview icon (just the generic quicktime icon).

[MacOS] Adding --extractor-arg "youtube:player_client=mediaconnect" to my script does provide the 95 format option, but when I run yt-dlp --list-impersonate-targets, I get the following response even though I've installed brew install cffi.
[info] Available impersonate targets
Client   OS   Source
---------------------------------------
Chrome   -    curl_cffi (not available)
Edge     -    curl_cffi (not available)
Safari   -    curl_cffi (not available)

I have Chrome and Safari installed on my laptop.
It's a bit of a shame that format 95 downloads very slowly, and sometimes does not provide a preview icon (just the generic quicktime icon).

FYI, 95 is used for the live broadcasts (and was replaced by 22 after some time, when these broadcasts are kept online afterwards).

even though I've installed brew install cffi

cffi is not the same as curl_cffi. yt-dlp also requires specifically curl-cffi==0.5.10. I don't think you can easily install this through brew. What browser you have installed does not matter.
To use the impersonate option, either use the static macos builds, as these include curl_cffi, or install yt-dlp and curl_cffi through pip/pipx.

It's a bit of a shame that format 95 downloads very slowly

adding this --concurrent-fragments 50 will be faster (use a smaller number if error occurred)

sometimes does not provide a preview icon (just the generic quicktime icon).

having ffmpeg installed may solve the icon problem
yt-dlp will automatically invoke ffmpeg to fix the mp4 after download

This is mainly a side effect of the android client being removed, due to server-side attestation being added by Google.  Format 22 is no longer reliably available.

Could you elaborate on the reliably part a bit?  Like, is it per-video or there's a way to force it somehow?

As of today, it seems that format 22 is gone for good: [...]

I'm wondering: can we, as a community, do anything about it?  Being able to download premuxed 720p videos had been hugely convenient; now I'm stuck with low-quality 360p (format 18) in FFmpeg-less environments.  How's that rational?!

Could you elaborate on the reliably part a bit? Like, is it per-video or there's a way to force it somehow?

No way to force it. It's up to youtube as to whether or not the format is served, and it is seemingly random whether or not 22 is served for some videos. (And the odds seem to be trending slimmer and slimmer.)

can we, as a community, do anything about it?

No

This is mainly a side effect of the android client being removed, due to server-side attestation being added by google. Format 22 is no longer reliably available. You will likely have to switch to 136+140 instead for reliable extraction. Do note that this requires ffmpeg for merging.

My issue with using the merge is rather odd.  I wrote a program that uses ffmpeg to re-encode/edit the (usually format 22) video from yt-dlp to add the video's thumbnail and title for a few seconds before the real video plays (better for playlist of videos for tv viewing).
I've noticed that recently when playing the edited video in Kodi (libreelec on a raspberry pi 3 hooked up to a tv) all is okay until I skip forward a few seconds, doing so sometimes temporarily shows a mostly green image, sometimes with a grid of tiny darker green squares.  The audio isn't affected.
I'm guessing that something is different between the merged video file (than the format 22).
I tested 3 unedited videos straight from yt-dlp, one that didn't have the 140 (used 140-4) and there were no seeking artifacts.
Could there be a muxing or some other difference between the "ffmpeg merging" and the youtube format 22 video?  I'm guessing I'll have to try adding some ffmpeg keyframe settings in my program.
The 140-4 stream is a "Good Mythical Morning" video with 5 language audio streams.

Many thanks to anyone who discovered this new issue.
Roughly a week ago, I noticed that youtube on kodi could not play anything but 360p when mpeg dash was disabled and today I had some time to try a few vids with yt-dlp -F, only to find out that format 22 is long gone.
And with that, my favorite mpv profile is to no use now, unless I switch it to format 18 for trouble free... 360p playback.
[clowns]
ytdl-format=22

Thank you very much @bashonly
Adding the lines:
-S "res:720,fps:30"
--extractor-arg "youtube:player_client=mediaconnect"

to my
.config/yt-dlp/config
seems to have resolved the issue 😂👍
(With mentioned options, I'm getting YouTube vids in 720p again with no need for curl_cffi nor the yt-dlp format option @pitsi @danfe )

You should still be able to get 720p without the mediaconnect client though; format 22 was just a pre-merged variant of 136+140. -S vcodec:h264,res:720,acodec:aac

When the issue arose, I changed:
-S "res:720,fps:30"
into the recommended:
-f 136+140
in my
.config/yt-dlp/config
And still got 360p for both streaming and downloading from YouTube.
Using:
-f 136+140
--extractor-arg "youtube:player_client=mediaconnect"

gives me 1920x1080 videos when streaming with mpv from YouTube, while format 136+140 is available for these videos.
It downloads the same videos at 720p though.
When using:
-S "res:720,fps:30"
--extractor-arg "youtube:player_client=mediaconnect"

I get 720p for both stream and download.
I used these .bashrc binds:
alias m='url=$(xclip -o -selection clipboard); echo "mpv $url"; echo -n "playing: "; yt-dlp --print title $url; echo; mpv --slang=en $url'
alias y='url=$(xclip -o -selection clipboard); yt-dlp -i --write-subs --sub-langs en $url'

when I tested the streaming and downloading.

mpv ignores any -f args in your config. (To be precise, it overrides them with its own -f arg.) And I didn't recommend using -f, I recommended:

-S vcodec:h264,res:720,acodec:aac

Also, -f 136+140 can't be used in conjunction with --extractor-arg youtube:player_client=mediaconnect

severely throttled

True, which is why mediaconnect's HLS formats are better than 136+140 for direct playback

Thanks to everyone here I also can now get 95 which is good enough for me!

Closing since there's nothing we can do

--extractor-arg "youtube:player_client=mediaconnect"

Today I've noticed that even format 18 (premuxed 360p) is no longer offered by default unless I specify this option (JFYI).

@danfe you are probably affected by https://github.com/yt-dlp/yt-dlp/issues/10608
Either update to nightly owle wait for the next release.

Update: the mediaconnect client now requires sign-in, and thus is no longer a viable workaround for many users who need a pre-muxed >=720p stream URL

Update: the mediaconnect client now requires sign-in, and thus is no longer a viable workaround for many users who need a pre-muxed >=720p stream URL

I imagine this is for some book-keeping on who's using up all the traffic. Is it known how much daily usage is tolerated before the account gets blocked? I have a "normal" usage, just off-site watching.

Confirming mediaconnect client cannot be used anymore. So there is no workaround anymore for pre-muxed >= 720p videos ?

Afaik. web_safari can still be used. Though all but the premerged formats require passing a potoken.

Thanks web_safari works as a replacement. Wondering for all longs since many clients seems to have been gated behind this "Please sign-in" barrier.


Cannot Submit an Adaptive Card Input Form

I can neither scroll down to view & fill out the other fields in this form, nor does anything happen upon clicking Submit.

Here is the JS function that builds the card object:
function registrationCardSchema(fullUserEmail) {
    return ({
          "contentType": 'application/vnd.microsoft.card.adaptive',
          "content": {
              "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
              "type": "AdaptiveCard",
              "version": "1.0",
              "body": [
                  {
                      "type": "ColumnSet",
                      "columns": [
                          {
                              "type": "Column",
                              "width": "auto",
                              "items": [
                                  {
                                      "type": "TextBlock",
                                      "text": "Skype TradeBlock Registration",
                                      "weight": "bolder",
                                      "size": "large"
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "We just need a few personal details to setup the TradeBlock account.",
                                      "isSubtle": true,
                                      "wrap": true
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "First name",
                                      "wrap": true
                                  },
                                  {
                                      "type": "Input.Text",
                                      "id": "firstName",
                                      "placeholder": "John"
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "Last name",
                                      "wrap": true
                                  },
                                  {
                                      "type": "Input.Text",
                                      "id": "lastName",
                                      "placeholder": "Smith"
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "Email",
                                      "wrap": true
                                  },
                                  {
                                      "type": "Input.Text",
                                      "id": "email",
                                      "placeholder"<EMAIL_ADDRESS>                                      "style": "email"
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "Phone Number"
                                  },
                                  {
                                      "type": "Input.Text",
                                      "id": "phone",
                                      "placeholder": "xxx.xxx.xxxx",
                                      "style": "tel"
                                  }
                              ]
                          }
                      ]
                  }
              ],
              "actions": [
                  {
                      "type": "Action.Submit",
                      "title": "Submit"
                      // "title": "Submit",
                      // "data": { "fullUserEmail": fullUserEmail }
                  }
              ]
          }
      }
    )
}

module.exports = { registrationCardSchema }

Unfortunately Skype does not support AdaptiveCards.  So what they do is render an image.  You could consider using regular cards.  Here is a link that may be helpful:
https://github.com/Microsoft/BotBuilder-Samples/tree/master/samples/javascript_nodejs/06.using-cards

@sgellock following up - also, where can I check to see which platforms do support adaptive cards? Cheers 😄


FR: Move some samples to /appengine

storage-xml-serviceaccounts-appengine
taskqueue-defer-sample
unittests-appengine-local-testing-samples

Notice: this issue has been closed because it has been inactive for 836 days. You may reopen this issue if it has been closed in error.


[ADAP-979] [Bug] Database Error: Only user perm tables supported in producer views

Is this a new bug in dbt-redshift?

[X] I believe this is a new bug in dbt-redshift
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
When the dbt Cloud connection is set to database "development" and your dbt_project.yml file states all model nodes occur in database sources then dbt compile and dbt build will not work.
Expected Behavior
The database specified in dbt_project.yml for a model is used as specified. At the least, when using RA3 nodes and a database user that has permissions, dbt should be able to run dbt compile successfully regardless of the database specified in the Connection.
Steps To Reproduce

Have a Redshift cluster with RA3 nodes. Have a database connection to database A. Have a Redshift db user with, at least, database permissions to database A and B. Have a dbt_project.yml file that specifies +database: B.
Run dbt compile.
Receive error message.

Relevant log output
22:19:23  Began running node model.redshift_us_east_1.clean_table_coconut
22:19:23  Re-using an available connection from the pool (formerly list_sources_dbt_developer_contracts, now model.redshift_us_east_1.clean_table_coconut)
22:19:23  Began compiling node model.redshift_us_east_1.clean_table_coconut
22:19:23  Using redshift connection "model.redshift_us_east_1.clean_table_coconut"
22:19:23  On model.redshift_us_east_1.clean_table_coconut: BEGIN
22:19:23  Opening a new connection, currently in state closed
22:19:23  Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
22:19:23  Redshift adapter: Connecting to redshift with username/password based auth...
22:19:23  SQL status: SUCCESS in 0.0 seconds
22:19:23  Using redshift connection "model.redshift_us_east_1.clean_table_coconut"
22:19:23  On model.redshift_us_east_1.clean_table_coconut: /* {"app": "dbt", "dbt_version": "1.5.9", "profile_name": "user", "target_name": "default", "node_id": "model.redshift_us_east_1.clean_table_coconut"} */
SELECT *
    FROM sources.information_schema.tables
    WHERE table_name = 'clean_table_coconut' and table_schema = 'dbt_developer_transforms'
22:19:23  Redshift adapter: Redshift error: Only user perm tables supported in producer views
22:19:23  On model.redshift_us_east_1.clean_table_coconut: ROLLBACK
22:19:23  Timing info for model.redshift_us_east_1.clean_table_coconut (compile): 22:19:23.485015 => 22:19:23.666072
22:19:23  On model.redshift_us_east_1.clean_table_coconut: Close
22:19:23  Database Error in model clean_table_coconut (models/upstream_prepared/sqlserverdbname_dbo/clean_table_coconut.sql)
  Only user perm tables supported in producer views
22:19:23  Finished running node model.redshift_us_east_1.clean_table_coconut

Environment
- OS: dbt Cloud
- Python: dbt Cloud
- dbt-core: 1.5
- dbt-redshift: dbt Cloud

Additional Context
Why can't I just change the database name in the dbt Cloud Connection to be correct? Because that will generate a new ssh key set which means the public_key has to get PRed by DevOps into the right place which will take an unspecified amount of time. (So yes, I'm going to go through that process, but also it would be great for this to be fixed so someone else doesn't have to deal with it. Also, supporting RA3 nodes in general.)
Why? Well it's not any of these:
A) user permissions. The Only user perm tables supported in producer views in the error message implies it's a permission issue. So I explicitly permissioned the user account for allllll the things - usage, select, all alter default privileges on tables, schema and added it to groups with at least SELECT permissions on the same. No go. This database user also created and owns the database it's referencing and all the objects it's trying to read information about. In addition, the database user in my dbt Cloud profile settings is a superuser. (Technically, it doesn't own this particular schema.)
B) It occurred to me that the dbt-redshift adapter should be doing something like USE DATABASE sources  but a) Redshift doesn't seem to have the USE keyword from a web search and b) dbt is correctly writing the database name sources in the query it runs against the information_schema.
C) Maybe it was that Redshift doesn't support cross-database work - unless you use RA3 nodes and then it supports SELECTs. These are RA3 nodes, so it should be able to run a SELECT on the sources database information_schema schema.
D) The database connection just doesn't work. Incorrect. i) dbt seed runs fine and creates 3 tables, albeit in the wrong place - database apple. This is expected - RA3 nodes don't support writes. (Specifying database B for seed writes also results, as expected, in an error.) ii)There's a SUCCESS message for the connection in the log.
E) Fluke one off. No, this happened multiple times.
G) Something about whether an object exists or not. I received the error whether the object I was trying to refresh already existed or didn't(deleted manually outside of dbt).
You can possibly argue what the bug is/bugs are, but something doesn't work correctly. Possible interpretations of this bug are:

Redshift itself has a problem.
dbt-core is missing some code/syntax around supporting RA3 nodes and/or cross-database reads.
dbt-core could return a better error message.
dbt Cloud Connection page could have a warning about what database name you put in.
User error. I'm completely missing something.
Something else.

Thanks for detailed write-up @alison985 !
I'm wondering if we can reproduce the essence of your experience outside of dbt Cloud. Are the following files a reasonable representation of the setup and experience you are describing?
Files
profiles.yml
sandcastle-redshift:
  target: connection_a
  outputs:

    connection_a:
      type: redshift
      dbname: A
      ra3_node: true
      # ...

    connection_b:
      type: redshift
      dbname: B
      ra3_node: true
      # ...

dbt_project.yml
name: "my_project"
version: "1.0.0"
config-version: 2
profile: "sandcastle-redshift"

models:
  my_project:
    +database: B

models/my_model
select 1 as id

Commands
This works?
dbt run -s my_model --target connection_b

But this doesn't?
dbt run -s my_model --target connection_a

Hi @dbeatty10 ! That file setup is correct for what I had except for the ra3_node: true and that I switched to testing with dbt compile since that theoretically has the least amount of database dependency.
I have been using dbt-core locally with this Redshift cluster for weeks, so I can do the outside dbt Cloud test.
Yes, it works fine with connection_b. (relevant database B in profile.yml)
Yes, it doesn't work with connection_a. (non-relevant database A in profile.yml) I get this error message locally:
04:16:43  Encountered an error:
ERROR: Cross-db references allowed only in RA3.* node. (sources vs data_platform_development)

When I add ra3_node: true with non-relevant database A in profiles.yml I get this error message:
Runtime Error
  Database Error in model clean_table_coconut (models/upstream_prepared/sqlserverdbname_dbo/clean_table_coconut.sql)
    Only user perm tables supported in producer views

When I keep ra3_node: true and switch back to relevant database B, it compiles successfully.
FWIW, locally I'm on dbt-core v1.4.8 and dbt-redshift v1.4.0. The dbt_project.yml file says require-dbt-version: [">=1.4.6", "<1.6.0"] which allows for local use of a SQL Server connection. Because dbt Cloud project is configured to 1.5, the same code runs on dbt v1.5 in the cloud.

Nice research @alison985
I haven't seen this error before and didn't have much luck with Google either, so we might need to ask some folks from AWS what it means:
    Only user perm tables supported in producer views

In the meantime, could you try installing dbt-redshift 1.7.0 and setup a brand new standalone project like this and see if you get that same error?

@dbeatty10 , I can't upgrade the code base to 1.7 because then I would lose the ability to function against SQL Server.

Could you help me understand how SQL Server and Redshift are related to each other in this case?
Is there some way by which you are connecting to a SQL Server database from within a single dbt project that uses the dbt-redshift adapter?
Or do you have both the dbt-redshift adapter and the dbt-sqlserver adapter installed at the same time with a shared version of dbt-core? And you have totally different dbt projects that use each of those adapters separately?
Virtual environment for dbt-redshift 1.7
Assuming that you have a separate dbt project that only uses Redshift (and doesn't use SQL Server at all), could you try to install 1.7 within a virtual enviornment like this? The details will vary based on your operating system and shell. If you let me know your operating system, I can try coming up with custom instructions for you.
These are the instructions for macOS using zsh or bash:
python3 -m venv redshift_1.7
source redshift_1.7/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade --pre dbt-redshift~=1.7.0.dev0 dbt-postgres~=1.7.0.dev0 dbt-core~=1.7.0.dev0
source redshift_1.7/bin/activate
dbt --version

👉 Then could you try setting up a brand new standalone project like https://github.com/dbt-labs/dbt-redshift/issues/656#issuecomment-1791715256 and see if you get that same error?
When you are all done using this virtual environment, you can get out of it by running:
deactivate

Determining your python command
You can run the following to find your python command that points to Python 3 (rather than Python 2):
python3 --version
python --version

If both of them say something like Python 3.x.x, then you can use either interchangeably.

dbt-sqlserver shouldn't have anything to do with it. I was trying to say I can't upgrade my dbt-core version until dbt-sqlserver supports dbt-core versions > 1.4.8. Or at least, I'm not going to attempt adding it to the complexity of the project.
I have 1 dbt code repo(and dbt_project.yml). Locally, within 1 Docker container of dbt-core I have dbt-sqlserver and dbt-redshift adapters. There are 2 different profiles in profiles.yml, one for each dialect. I can switch back-and-forth through environment variables. I've done enough abstraction, macros, variables, and changes that the dbt code will run on either dialect.
Whether using the Redshift adapter and Redshift development endpoint on a) dbt-core v1.4.8 locally and/or b) dbt Cloud v1.5 with 1 dbt Cloud project, dbt can not handle reading across databases on RA3 nodes for at least some operations(e.g. compile).

Assuming that you have a separate dbt project that only uses Redshift (and doesn't use SQL Server at all),

I don't. I was already setup to do the outside-dbt-Cloud test you requested and I was happy to do it. Unfortunately, I won't have time to create the separate project test you're describing.
I can report that dbt Cloud works now that the database name in the connection string is changed to the database name I'll be working in and the dbt Cloud generated public key is installed on our side.

Hi @alison985 , looks like you're trying to run a dbt project with models on a database while the connection is made to another database. Is that correct? You're right that RA3 nodes support SELECTs on another database, but CREATEs are not supported. I think the issue here is that the models being run in your project will attempt to run CREATE TABLE or CREATE VIEW on database B while connected to database A. Your project will be able to doSELECTs in database B using database A connection, but cannot do write operations. This is also why dbt seed created tables in database A. If you only want to read from database B using connection_a, consider using other dbt objects such as a macro that won't attempt to create anything in the unconnected database.

Ah, thanks for the clarification @dbeatty10 . I did an experiment querying on a ra3 cluster using redshift query editor. I ran the following snippet while connected to cluster_b and got the same "perm tables" error.
SELECT *
   FROM cluster_a.information_schema.tables

This may be a limitation with cross-database querying information_schema, guessing that it is not accessible from another database. An alternative would be to use svv_redshift_tables. Try the following:
SELECT *
    FROM svv_redshift_tables
    WHERE database_name  = cluster_a

The query above should return a list of all tables in cluster_a.

Thanks for that insight @jiezhen-chen !
Potential fix
Searching through the source code, the fix might be updating here (and maybe here too) to use svv_redshift_tables like you mentioned.
I'm not sure one way or the other if there are other queries against information_schema.tables, etc. that dbt-redshift has directly or inherits indirectly from dbt-postgres.
Acceptance criteria

The following test case breaks without the fix, but works after the fix

Test case details
Files
profiles.yml
sandcastle-redshift:
  target: connection_a
  outputs:

    connection_a:
      type: redshift
      dbname: database_a
      ra3_node: true
      # ...

    connection_b:
      type: redshift
      dbname: database_b
      ra3_node: true
      # ...

dbt_project.yml
name: "my_project"
version: "1.0.0"
config-version: 2
profile: "sandcastle-redshift"

models:
  my_project:
    +database: database_b

models/my_table.sql
{{ config(materialized="table") }}

select 1 as id

models/my_view.sql
{{ config(materialized="view") }}

select 1 as id

models/my_mv.sql
{{ config(materialized="materialized_view") }}

select 1 as id

models/my_ephemeral.sql
{{ config(materialized="ephemeral") }}

select 1 as id

Commands
This is expected to break until a fix is in place:
dbt run --target connection_a

This is expected to work just fine both with and without a fix:
dbt run --target connection_b

[!NOTE]
I didn't actually try reproducing this error report myself.

Will likely be addressed as a part of https://github.com/dbt-labs/dbt-redshift/issues/555

Lacking some context as I am not a DBT user but from what I can tell Redshift does not support cross DB queries on some system level tables...


TextField, using the scanner device, can't recognize the return key automatically added by the system, and the onChange method can't detect the newline. . . ,

TextField, using the scanner device  can not identify the system automatically added the Enter key, the onChange method also does not detect line breaks. . . Is there any good way to solve this problem?
Scanner device is android 5.0 system

I have the exact some issue but on Android 6.0

What method did you use to solve this problem?

Could not solve it, i still have the issue.
I commented to bump it.

@justinmc  Is there any temporary solution to this problem?

Not that I can think of...  When does the system add a line break automatically?  I'll try to reproduce this.

@justinmc ，Thank you for your reply。
I am using the flutter to implement the function of entering barcodes into the application using a barcode scanner device.
The barcode scanner device can append a line break.
Flutter‘s TextFeild can get scanned barcodes, but can't handle line breaks.
On the android system, it can be processed by setOnEditorActionListener on the EditText.
I look forward to your giving you a good way.
Thank you.

I don't think I can reproduce this.  I can definitely help if you want to dig into this yourself though.
You might want to look at what TextEditingValue is coming from the engine when a problematic newline is entered:
https://github.com/flutter/flutter/blob/4e3698fb5dcbf8a88135403a9ac19e2b6b3f39ac/packages/flutter/lib/src/services/text_input.dart#L655-L661
If it looks correct at that point, you'll have to compile the engine locally and try to check what's coming from the device.

@jicg, were you able to answer @justinmc';s question? Is this still happening with the latest stable release?

I can't answer this question, I don't know how to compile the local engine, this is too difficult for me. I tested this issue and it still exists in the latest version

I have used native development to solve the problem.

Hi @jicg
Since you have moved and issue  reproducible without any additional information
given your last message I feel safe to close this issue, if you disagree please write in the comments and I will reopen it.
Thank you


Rename parallel feature to rayon.

This is now possible because of namespaced dependencies, introduced in rustc 1.60.0.
This PR contains a temporary workaround for trybuild issue #171.

Codecov Report

Merging #81 (514d07d) into master (9f87ea3) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #81   +/-   ##
=======================================
  Coverage   40.16%   40.16%
=======================================
  Files          56       56
  Lines        5920     5920
=======================================
  Hits         2378     2378
  Misses       3542     3542

Impacted Files
Coverage Δ

src/archetype/mod.rs
10.50% <ø> (ø)

src/archetypes/mod.rs
49.65% <ø> (ø)

src/archetypes/par_iter.rs
0.00% <ø> (ø)

src/query/result/par_iter.rs
0.00% <ø> (ø)

src/query/view/assertion_buffer.rs
100.00% <ø> (ø)

src/system/mod.rs
0.00% <ø> (ø)

src/system/null.rs
0.00% <ø> (ø)

src/system/par.rs
0.00% <ø> (ø)

src/system/schedule/mod.rs
0.00% <ø> (ø)

src/world/mod.rs
0.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Rerunning the miri job. I want to know whether it's flaky (which is not good) or if it's just something that is present in the branch but not in the actual pull to master.

Hmm. Looks like miri passes now, which is actually worse than if it had failed again. There may be something fishy in the Schedule code.
For reference, this is the failing job: https://github.com/Anders429/brood/runs/7757239434?check_suite_focus=true
Anyway, this doesn't need to block this PR. Opened #82 to track that issue separately.


Auth invalid password when using MD5 password

Just hit this issue recently on Vertica Community Edition 6.0.1, fairly confident that the problem will exist with Vertica Enterprise as well.
It's easy to repro, just set authentication on server to MD5, then try to use the driver.
Workaround for now is to set vertica authentication to password.
I'll try to deal with this issue this weekend.

It is not clear how to use ssl param at all. Got error message
Error:<PHONE_NUMBER>37408:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1487:SSL alert number 40
at Error (native)
at SlabBuffer.use (_tls_legacy.js:39:18)
at CleartextStream._read (_tls_legacy.js:257:29)
at CleartextStream.Readable.read (_stream_readable.js:348:10)
at EncryptedStream._write (_tls_legacy.js:171:25)
at doWrite (_stream_writable.js:334:12)
at writeOrBuffer (_stream_writable.js:320:5)
at EncryptedStream.Writable.write (_stream_writable.js:247:11)
at Socket.ondata (_stream_readable.js:555:20)
at emitOne (events.js:96:13)


chore: add integration tests

Issue
Currently there are no integration tests to check a plugin update with the reaction version
Solution
This PR adds a circle CI step that runs the integration tests on the plugin.
Breaking changes
None expected
Testing

Go to the CircleCI jobs, in the logs you can see the integration tests being run.

:tada: This PR is included in version 1.0.7 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Project cleanup

Hey there,
I decided to use this date picker in our project and found a bug.  There where a lot of errors while running npm run installand npm run start-dev. So we fixed some stuff, hope we could help others too!
The bug was btw just on the current npm release 0.9.4. Everything worked fine with 0.9.3 and also worked using this repo directly! Maybe there went something wrong while building. Hope you can create a new release soon! :)
While fixing the errors I found some things I fixed (mostly semicolons).
Hope this can be merged (should not break anything)!
If you like I would also like to update the README file to contain more information about what which parameter does.
I would also recommend using a linter for this project, but this should maybe discussed in an issue ;)
Kind regards
Alex

Hi @Pingu501 thank you for pr
linter added, bugs resolved, react moved to peer deps and the new version released at next tag.


Use Pow.PasswordReset.Plug.create_reset_token without a conn

I have a site (https://github.com/axelclark/golf) that has both an html frontend and a graphql api for a mobile app.  I have the ResetPassword extension set up for html.  I'm trying to enable a user to send the reset token email from their mobile app using graphql.
I'm trying to figure out which Pow.PasswordReset functions to use in my graphql resolver which acts like a controller.  The problem is I don't have a conn with the config in it.  I just have the user email from the params.
  def create_reset_token(_parent, %{email: email}, _resolution) do
    # Need to create the token and send the email
  end

Any suggestions on how to do this?  I only want to enable the create action through graphql.

Here is my current approach which seems to be working.  I send the email address back regardless of whether the email is sent.
defmodule GolfWeb.Resolvers.Accounts do
  alias GolfWeb.Router.Helpers, as: Routes
  alias PowResetPassword.Phoenix.Mailer

  def create_reset_token(_parent, %{email: email_address}, _resolution) do
    config = Application.get_env(:golf, :pow, [])
    conn = Pow.Plug.put_config(%Plug.Conn{}, config)
    params = %{"email" => email_address}

    conn
    |> PowResetPassword.Plug.create_reset_token(params)
    |> respond_create(email_address)
  end

  defp respond_create({:ok, %{token: token, user: user}, conn}, email_address) do
    url =
      Routes.pow_reset_password_reset_password_url(
        GolfWeb.Endpoint,
        :edit,
        token
      )

    email = Mailer.reset_password(conn, user, url)
    Pow.Phoenix.Mailer.deliver(conn, email)

    {:ok, %{email: email_address}}
  end

  defp respond_create({:error, _, _}, email_address) do
    {:ok, %{email: email_address}}
  end
end

I found this closed issue when trying to wire up login/registration via absinthe. While this is super helpful, I am running into problems when attempting to use this method. Is there something special about session/registrations, by any chance?
        %Plug.Conn{}
        |> Pow.Plug.put_config(otp_app: :kyodo_web)
        |> IO.inspect()
        |> Pow.Plug.authenticate_user(%{"email" => input.email, "password" => input.password})

Results in:
# Pow plug was not found in config. Please use a Pow plug that puts the :plug in the Pow configuration.
I'm aiming to move all of this functionality into Absinthe and then I'd be into making a pow-absinthe package if things go well... Either way, maybe I can help w/ the guides if there are some good learnings.

In case this helps someone, I was trying to figure out how to deliver an invitation mail from a LiveComponent, and this issue put me on the right track i.e.:
  defp deliver_invitation_email(socket, user, invited_by) do
    conn = conn(socket)
    token = PowInvitation.Plug.sign_invitation_token(conn, user)
    url = Routes.pow_invitation_invitation_url(conn, :show, token)
    email = Mailer.invitation(conn, user, invited_by, url)

    Pow.Phoenix.Mailer.deliver(conn, email)
  end

  defp conn(%{endpoint: endpoint} = _socket) do
    Conn
    |> struct!(%{
      private: %{phoenix_endpoint: endpoint},
      secret_key_base: endpoint.config(:secret_key_base)
    })
    |> Pow.Plug.put_config(otp_app: :my_app)
  end

I just want to note, that it isn't a good idea to pass conn to absinthe context and rely on it because GraphQL API is transport agnostic. In my opinion building new conn inside a resolver is the best workaround nowadays.

This worked for me, though this line
defp conn(), do: Pow.Plug.put_config(%Plug.Conn{}, otp_app: :golf)

is giving me dialyzer errors about success typing and the contract. If I had to guess based on previous dialyzer battles, this is because the Plug.Conn typespec has keys which need a value (like pid), and the argument we are passing is primarily empty
I suppressed dialyzer for that file to make progress

@dlobo good call, should probably do something similar to Plug.Adapters.Test.Conn: https://github.com/elixir-plug/plug/blob/f24af6a5b00dba4d32d44c6f2fb4423f6a5d8e3b/lib/plug/adapters/test/conn.ex#L34-L50

In my case, the Pow calls just needed the config object (basically I wanted to delete the existing tokens in case the user updated the roles, so deleting the token in persistent and credentials cache)
so I just sent in the config as: [otp_app: :glific] and did not need the conn object (and hence did not need to get in via absinthe either)

I am still getting this error "Pow plug was not found in config. Please use a Pow plug that puts the :plug in the Pow configuration."
config = Application.get_env(:myapp_api, :pow, [])
conn = Pow.Plug.put_config(%Plug.Conn{}, config)
def signin(conn, attrs \ %{}) do
user = pow_authenticate(%{"email" => attrs.email, "password" => attrs.password})
if user != nil && user.is_disabled do
  nil
else
  {result, connection} = conn
  |> Pow.Plug.authenticate_user(%{"email" => attrs.email, "password" => attrs.password})

  {result, connection, user}
end

end


Hotfix ZeroconfServer not compiling

ZeroconfServer Doesn't compile with StandardCharsets.UTF_8 in Java 11.
java: incompatible types: java.nio.charset.Charset cannot be converted to java.lang.String.
URL parameters should be of String type.

IntelliJ is a bit stupid, it changes stuff without asking.


Error $customConfig cannot be retrieved because it has not been set

When creating a container with -includeCSide and -shortcuts None the script throws the following error: The variable '$customConfig' cannot be retrieved because it has not been set.
This happens because the custom settings are only read when shortcuts -ne "none".
New-NavContainer.ps1 line 401 - customsettings are read.
New-NavContainer.ps1 line 443 - $customConfig is used without a read.
This occurs on the latest version (<IP_ADDRESS>)

Thanks - will fix this asap

Thanks :)

Fixed in <IP_ADDRESS>


Add timeout for getting the next record when sampling

Description
Previously, when sampling streaming data, the sampler would wait timeoutMs for numRows to be returned. If the stream has less rows than numRows, the sampler would wait the full timeoutMs period. In some cases, initially connecting to a given stream to sample may take a considerable amount of time, but reading subsequent records thereafter is a much faster operation. For these cases, this PR adds a new sampler config property nextRecordTimeoutMs, which if specified, causes the sampler to return after this period of time if no further records are found between the last record found, or timeoutMs, whichever comes first.  The new nextRecordTimeoutMs is only applied if at some records have been read from the stream during sampling. For example if a stream has only 1 record in it at a given time, and timeoutMs is configured as 15000 miliseconds and nextRecordTimeoutMs is configured as 2000 milliseconds while numRows is configured as 500, then assuming the first record is read quickly, the sampler will return after roughly 2000 milliseconds, whereas previously it would have returned after the full 15000 millesecond timeoutMs config value.
This PR has:

[x] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

We really have to resist the temptation to add a config for every corner case we run into. Is it absolutely necessary to have this config? Even if you need these, why not use standard terms such as connectTimeout, readTimeout that are more common names and easier to relate to than something like nextRecordTimeout?

We really have to resist the temptation to add a config for every corner case we run into. Is it absolutely necessary to have this config? Even if you need these, why not use standard terms such as connectTimeout, readTimeout that are more common names and easier to relate to than something like nextRecordTimeout?

@abhishekagarwal87 , yes I can change to readTimeout, that is a better name.

I am not entirely sure we need a config here either. Irrespective of what we name the config, it is going to cause confusion with the existing timeoutMs, especially since the SamplerConfig is used with batch too where this config would not be meaningful.
I think we could just have a fixed maximum number of empty polls. When we poll the topic for records, and say we don't get anything for 10 tries or say, we just wrap up the sampler.
Since this is only used for sampling, there is no real obligation to keep waiting until we have read the requested number of records. Compare this to batch where if there is no file at the specified path, we just fail or return empty and don't wait for data to show up.
@zachjsh , @abhishekagarwal87 , what do you think?

I am not entirely sure we need a config here either. Irrespective of what we name the config, it is going to cause confusion with the existing timeoutMs, especially since the SamplerConfig is used with batch too where this config would not be meaningful.
I think we could just have a fixed maximum number of empty polls. When we poll the topic for records, and say we don't get anything for say 10 tries, we finish the sampling.
Since this is only used for sampling, there is no real obligation to keep waiting until we have read the requested number of records. Compare this to batch where if there is no file at the specified path, we just fail or return empty and don't wait for data to show up.
@zachjsh , @abhishekagarwal87 , what do you think?

thanks @kfaraz , just wanted to be clear that this new config is not strictly needed for things to work properly, most users will not have to touch this. However its usage enabled an improved UX
About your idea described above, the poll timeout is hardcoded to 100 milliseconds. There could be cases where a user may want to allow for longer than 1 second to for the next record to be read. As for confusion with timeoutMs, let me know if there is something I can add that helps differentiate them

I feel that some kind of config here makes sense, either as a timeout or a max number of tries after encountering an empty read, to allow the behavior to be disabled or adjusted to the characteristics of a specific stream
imo it should be called something like streamingNextRecordTimeout or something with "streaming" in the name if we're concerned that there would be confusion with batch sampling cases

It seems that the intent here is to prevent sampling from Kafka getting stuck when fewer than numRows exist in the stream. Instead of adding a second shorter timeout, could we tackle that problem directly? Something like having the sampler check the latest offset available in Kafka first (using RecordSupplier#getLatestSequenceNumber) and then returning early once it has read up through that latest message.

Btw, on general design. Sampling is intended to be used in a very specific situation— it's there so the web console (& any similar third-party UIs) can show people a few records from a stream prior to configuring their supervisors. I can't imagine that most users will set any of these config parameters in a case like that. They'll just get whatever behavior the UI requests from the API.
For that reason I think it's important to focus on the out-of-box behavior, so users using the builtin web console get the benefit of these adjustments.

Something like having the sampler check the latest offset available in Kafka first (using RecordSupplier#getLatestSequenceNumber) and then returning early once it has read up through that latest message.

Would this cause issues when useEarliestOffset is false? Or if the earliest offset happens to be close to the latest in a stream that's actively receiving data?
From my understanding, the timeout here is basically defining how long a stream can be idle (have no data arrive / offsets stay the same). The goal is to improve the experience for web console (and similar UIs) when a stream is low volume at the time ingestion is being setup in Druid. Right now, for the web console, if there are fewer than 500 (numRows) rows, each call to the sampler always takes 15 seconds (timeoutMs). The web console calls the sampler a lot during the normal flow, so this can be pretty slow.

I can't imagine that most users will set any of these config parameters in a case like that.

Agreed. The intent of the config in the sampler payload is to allow UIs to tweak their settings without users having to redeploy Druid. It's not intended to be set (or even seen) by normal end users.


Remove WorkflowLogger

This PR decouples logging logic and exception reporting logic.

Remove WorkflowLogger, decoupling its logic into logger and the FatalError control message. Now log error will not necessarily send FatalError.
Add AmberLogging trait, to provide a logger:

Each class now has its own logger, created with name [actorId] [class.simpleName].
The actorId: ActorVirtualIdentity is assigned when creating the Actor, and then passed to other classes used by this Actor.
All loggers are using SLF4j Facade, will use logback as the backend.

Remove WorkflowRuntimeError.
Redesign WorkflowRuntimeException to a more standard way of extending java.lang.RuntimeException.
Bind WorkflowRuntimeException to be logged as error. On a separate logic, the FatalError control message is propagated to the frontend to display.
Keep operator logic errors in LocalOperatorException control message to be propagated to the frontend for display.

The current way of attaching ActorVirtualIdentity cannot be extended to the internal objects, such as operators running inside the worker. We need to find a way to do this.

The current way of attaching ActorVirtualIdentity cannot be extended to the internal objects, such as operators running inside the worker. We need to find a way to do this.

We will discuss those internal object's logging later.


Add expire chain element and add tests to cover

Signed-off-by: denis-tingajkin<EMAIL_ADDRESS>

Blocked by https://github.com/networkservicemesh/sdk/pull/356


Simple query causing massive memory usage (pushing over the half gigabyte level allowed by Heroku)

This is an issue that I'm digging into - I'm tracking my findings here so I can ask for help later if necessary.
The minimal query to trigger this is:
{
  sketchById(id: 11) {
    sketchIntegrationEventRunsBySketchId(orderBy: ID_DESC, first: 100) {
      edges {
        node {
          __id
        }
      }
    }
  }
}

So I've downloaded the data from my staging environment and am running the dev version of PostGraphQL locally. I have 16GB of RAM whereas Heroku only gives 512MB so it's not a very fair comparison, and I have to monitor the memory very fast to see the issue.
In order to do so, I've set up this loop (OSX, so using gdate from coreutils):
while sleep 0.001; do
  echo -n "`gdate +%s%3N` ";
  ps -o 'pid=,rss=' -p 45416;
done | tee log.txt

This has given me quite a high resolution file exposing the memory usage. I'm going to build a graph from it and upload... cat log.txt | sed 's/ /'$'\t''/g' > memory.tsv

I ran the query six times; first with gaps of a few seconds between each one, finally in quick succession. The x-axis is time in milliseconds, y-axis is RSS memory usage in kilobytes.
As can be seen, this simple query is allocating a hell of a lot of RAM.

I've now added --inspect to the ts-node command line and connected Chrome Canary as a debugger to the node process. I ran the query a few times to "warm up" the system; then used the "Record Allocation Profile" profiling tool for a single query execution to see where the RAM is being allocated:

Setting DEBUG="*" I get:

  postgraphql:request GraphQL query request has begun. +7s
  body-parser:json content-type "application/json" +11s
  body-parser:json content-encoding "identity" +2ms
  body-parser:json read body +1ms
  body-parser:json parse body +4ms
  body-parser:json parse json +0ms
  body-parser:urlencoded body already parsed +0ms
  body-parser:text body already parsed +1ms
  postgraphql:request GraphQL query is parsed. +13ms
  postgraphql:request GraphQL query is validated. +10ms
  postgraphql:graphql { sketchById(id: 11) { sketchIntegrationEventRunsBySketchId(orderBy: ID_DESC, first: 100) { edges { node { __id } } } } } +1ms
  postgraphql:postgres begin +1ms
  postgraphql:postgres select row_to_json(__local_0__) as object from "hookhaven_data"."sketch" as __local_0__ where "id" = any($1) limit $2 +8ms
  postgraphql:postgres select to_json(__local_0__) as value from "hookhaven_data"."sketch_integration_event_run" as __local_0__ where "id" is not null and true and true and ("sketch_id" = $1) order by "id" using > limit $2 +6ms
  postgraphql:postgres:explain
Limit  (cost=0.00..2.14 rows=1 width=32) (actual time=0.129..0.129 rows=1 loops=1)
  ->  Seq Scan on sketch __local_0__  (cost=0.00..2.14 rows=1 width=32) (actual time=0.128..0.128 rows=1 loops=1)
        Filter: (id = ANY ('{11}'::integer[]))
        Rows Removed by Filter: 3
Planning time: 0.052 ms
Execution time: 0.142 ms +1ms
  postgraphql:postgres commit +2s
  postgraphql:postgres:explain
Limit  (cost=222.36..222.59 rows=90 width=36) (actual time=419.363..438.374 rows=90 loops=1)
  ->  Sort  (cost=222.36..222.59 rows=90 width=36) (actual time=419.362..438.358 rows=90 loops=1)
        Sort Key: id DESC
        Sort Method: external merge  Disk: 69008kB
        ->  Bitmap Heap Scan on sketch_integration_event_run __local_0__  (cost=4.98..219.44 rows=90 width=36) (actual time=1.892..145.971 rows=90 loops=1)
              Recheck Cond: (sketch_id = 11)
              Filter: (id IS NOT NULL)
              Heap Blocks: exact=80
              ->  Bitmap Index Scan on index_sketch_integration_event_run__fk_sketch_id  (cost=0.00..4.96 rows=90 width=0) (actual time=0.019..0.019 rows=90 loops=1)
                    Index Cond: (sketch_id = 11)
Planning time: 0.088 ms
Execution time: 509.332 ms +219ms
  postgraphql:request GraphQL query has been executed. +1ms
  postgraphql:request GraphQL query request finished. +2ms
0 error(s) in 2487.72ms :: { sketchById(id: 11) { sketchIntegrationEventRunsBySketchId(orderBy: ID_DESC, first: 100) { edges { node { __id } } } } }

The important part of that seems to be:
  postgraphql:postgres begin +1ms
  postgraphql:postgres select row_to_json(__local_0__) as object from "hookhaven_data"."sketch" as __local_0__ where "id" = any($1) limit $2 +8ms
  postgraphql:postgres select to_json(__local_0__) as value from "hookhaven_data"."sketch_integration_event_run" as __local_0__ where "id" is not null and true and true and ("sketch_id" = $1) order by "id" using > limit $2 +6ms
  postgraphql:postgres:explain
Limit  (cost=0.00..2.14 rows=1 width=32) (actual time=0.129..0.129 rows=1 loops=1)
  ->  Seq Scan on sketch __local_0__  (cost=0.00..2.14 rows=1 width=32) (actual time=0.128..0.128 rows=1 loops=1)
        Filter: (id = ANY ('{11}'::integer[]))
        Rows Removed by Filter: 3
Planning time: 0.052 ms
Execution time: 0.142 ms +1ms
  postgraphql:postgres commit +2s

The commit (last line) didn't fire until 2 seconds after the explain?!

Running a CPU profile over a single query yields:

Quite a lot of this time is spent in the pg module itself.

So running the second query returns 72MB of data.
$ echo 'select to_json(__local_0__) as value from "hookhaven_data"."sketch_integration_event_run" as __local_0__ where "id" is not null and true and true and ("sketch_id" = 11) order by "id" using > limit 100' | time psql bots_staging | wc -c
 72609787
psql bots_staging  6.36s user 0.13s system 75% cpu 8.646 total
wc -c  0.29s user 0.04s system 3% cpu 8.645 total

So it seems that the issue here could be fixed along with #265: basically we're requesting data that we don't need, and if that data is large then it can cause massive performance/memory allocation issues.

This issue seems to match what you're hitting: https://github.com/brianc/node-postgres/issues/1103
How many fields are you selecting? This may be fixable by not using * and only getting the fields you need.

 It's high priority so I'll see what I can do.

I've been looking into #265 for a while; it seems this is a good starting point. Basically we want to remove the createPgProcedureObjectTypeGqlFieldEntry workaround, and instead do something like:
resolve(source, args, {fieldASTs}) {
  // Look up SQL columns (and computed columns!) required by fieldASTs
  // and add them to the select right here
}

which instead of doing select row_to_json(${sql.identifier(aliasIdentifier)}) as object would do something more like
select json_build_object('col1', ${tbl}.col1, 'col2', ${tbl}.col2, 'computedField1', computedField1(${tbl}), 'computedField2', computedField2(${tbl}, 27, 'Foo')) only referencing the requested columns.
I've not yet dug into fieldASTs so I'm not sure how transparent/opaque it is.

@calebmer Why are we doing row_to_json(...) rather than merely returning * directly - were you expecting to add more "meta" columns later, or maybe it's for compatibility with the _hooks that send the data back to postgresql later?

(Using psql, replacing to_json(__local_0__) with __local_0__.* cuts the query time from ~8 seconds to ~6.8, so it's only about 15% faster - I don't think that's worth making the code changes for.)

We use row_to_json for easier composite type support. row_to_json will turn composite types into nested objects which is convenient.
The first fix I want to implement is just selecting the columns that are requested when there are no procedures. So for now at least a procedure column will cause a de-opt as we revert to the * case. Of course in the future I want to implement the solution we discussed in #265 for procedures, but for now I’m at least focusing on the more common case which is at least a little easier to fix. Does the query that ends up breaking your Heroku box use procedure columns?

I use computed columns heavily, but fortunately not in this one case. (I do have performance issues related to computed columns, but they're not as problematic as this one is.)

Ok. Computed columns are definitely something I really want to make fast as well given how great they are for the developer experience. Out of curiosity do you use a lot of connection computed columns? Those will be pretty tricky to optimize and may always require a de-opt.

I'm going to keep taking notes here as I dig further into understanding the codebase (something I've been meaning to do for a while).

Using the awesome ts-node --inspect I've added a debugger statement at the top of readPage in PgPaginatorOrderingAttributes; and am now stepping through the backtrace because I can't find the fieldASTs property. Here's how the data looks as it leaves the graphql module and enters postgraphql:

Digging deeper into the info object, we can see the fields that are being requested on sketchIntegrationEventRunsBySketchId; ultimately our {edges: {node: { __id}}} path:

So this is probably what fieldASTs has become?

Confirmed: in v0.8.0 fieldASTs was renamed to fieldNodes:
https://github.com/graphql/graphql-js/releases

Awesome :+1:
I will note the approach I’m taking here. Because the GraphQL and Postgres parts of the codebase are kept separate (and intentionally so) I’m going to add a optimizations object which the GraphQL adapter can pass through the interface. The optimizations object will contain all of the fields selected in GraphQL. My current designs for an optimizations object depend on the changes in https://github.com/calebmer/postgraphql/pull/281. So I’m rushing to finish that first.
Out of curiosity, does fieldNodes resolve fragment fields? Or will we have to resolve fragment fields?

Looks like resolve takes 4 arguments: source, args, context and info; and it's info.fieldNodes that we need.

Good question, I shall investigate...

New query, with an inline and an external fragment and a relation lookup:
{
  sketchById(id: 11) {
    sketchIntegrationEventRunsBySketchId(orderBy: ID_DESC, first: 100) {
      edges {
        node {
          __id
          ... on SketchIntegrationEventRun {
            createdAt
          }
          ... F0
        }
      }
    }
  }
}
fragment F0 on SketchIntegrationEventRun {
  id
  sketchBySketchId {
		...F1
  }
}
fragment F1 on Sketch {
  updatedAt
}

It seems it does not lookup the fragments for you, you'll need to resolve them yourself:

Looks like operation is the entire query, whereas fieldNodes is specifically just the current thing that's being resolved.

Unfortunately I timed out on this. I’m not sure when I’m going to get the chance to work on it again. In order to solve some of the major issues like this one I’m going to need more community support. Either fiscally or in code contributions.

I'm guessing this is fixed in v4 so I'm going to tag it as so - if I accidentally close this without confirming please let me know!

#nostalgia


Store matched routes in request

@dougwilson I've been working on this feature (#2501).
It allows to store every matched route in a request when travelling through nested routes. This may be useful for logging, ACL checks, etc.
So, for example, if you have your routing set like this:
var fooRouter = new Router();
var barRouter = new Router();
var bazRouter = new Router();

bazRouter.get(['/bez', '/baz/:subId'], function(req, res, next) {
  next();
});

fooRouter.use(['/foo/:id', '/foe'], barRouter);
barRouter.use(['/bar'], bazRouter);

A request for the URL /foo/10/bar/baz/30 will set an array of matched routes in the request object like this:
req.matchedRoutes = ['/foo/:id', '/bar', '/baz/:subId'];

Let me know if there is anything that needs to be modification, unmeaningful variable names, etc.

Hi @ajfranzoia, I do have some concerns regarding the implementation, but I am not currently involved in this repository (you can see the lack of "Owner" or "Collaborator" by my name above). I'm only responding because you mentioned me specifically in this pull request.
I'm sure someone will stop by with feedback for Express, or you are always welcome to make this pull request against a module I am involved in, for example https://github.com/pillarjs/router
Sorry for the inconvenience during this ownership dispute, but I didn't want to leave you hanging since you mentioned me specifically.

Sorry for mentioning you, my mistake.
I will create a similar PR on your the pillarjs/router if you think this could be a feature that could be added.

@ajfranzoia ... nice! was just asked about this kind of thing at a recent node meetup. good stuff :-)

Hi @ajfranzoia, thank you for your pull request!
Unfortunately this pull request itself is not in a state that would be backwards-compatible with Express 4, which is the code your pull request is targeting. Express has been so API stable for such a long time, it's pretty much impossible to change any public API surface, or what seems like non-public API surface without breaking lots of production apps. Of note, it removes the .regexp and .key properties from all route objects. A lot of third-party code walks these objects to build route lists, etc.
The actual idea seems good, but should be made against 5.0, which does not even have route code. If you want, please transplant this pull request to see https://github.com/pillarjs/router for the discussion to open up there, where other router enthusiasts are as well :)

Thanks for your reply @dougwilson. I will definitly create a PR against https://github.com/pillarjs/router during the next days.


Prevent undefined behavior in QUICHE logging

Commit Message:
This change adds a NullGuard which will prevent us
from sending null strings to output streams to avoid
undefined behavior.
Signed-off-by: David Schinazi<EMAIL_ADDRESS>Additional Description: None
Risk Level: Low
Testing: Local
Docs Changes: None
Release Notes: None
Platform Specific Features: None
Fixes #15620

Looks like the gcc failure might be real - mind checking out CI?