Sample project does not work with Django 2.0.9

Django==2.0.9
django-js-asset==1.1.0
django-mptt==0.9.1
-e git://github.com/caktus/django-treenav.git@8f81619a8598790d1c2dc7bf77ba9d8e9e9564e6#egg=django_treenav
pytz==2018.7

!   ³ django-treenav  /d/w/g/django-treenav   develop  sample_project  ./manage.py  migrate
Traceback (most recent call last):
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/urls/conf.py", line 17, in include
    urlconf_module, app_name = arg
ValueError: too many values to unpack (expected 2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/base.py", line 332, in execute
    self.check()
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/base.py", line 364, in check
    include_deployment_checks=include_deployment_checks,
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 58, in _run_checks
    issues.extend(super()._run_checks(**kwargs))
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/management/base.py", line 351, in _run_checks
    return checks.run_checks(**kwargs)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/checks/registry.py", line 73, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/urls/resolvers.py", line 399, in check
    for pattern in self.url_patterns:
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/utils/functional.py", line 36, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/urls/resolvers.py", line 540, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/utils/functional.py", line 36, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/urls/resolvers.py", line 533, in urlconf_module
    return import_module(self.urlconf_name)
  File "/data/venvs/django-treenav/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/data/work/github/django-treenav/sample_project/urls.py", line 8, in <module>
    url(r'^admin/', include(admin.site.urls)),
  File "/data/venvs/django-treenav/lib/python3.5/site-packages/django/urls/conf.py", line 27, in include
    'provide the namespace argument to include() instead.' % len(arg)
django.core.exceptions.ImproperlyConfigured: Passing a 3-tuple to include() is not supported. Pass a 2-tuple containing the list of patterns and app_name, and provide the namespace argument to include() instead.

What is your urls.py?
It should look like this:
from django.contrib import admin
from django.urls import path, include

admin.autodiscover()

urlpatterns = [
    path('admin/', admin.site.urls),
    path('treenav/', include('treenav.urls')),

Sorry for the delay, but @itsdkey is now working on improving the sample project to support newer Django versions in #78


Validate the category in OSSL_trace_end()

OSSL_trace_end() should validate that the category it has been passed
by the caler is valid, and return immediately if not.
Fixes #17353

24 hours has passed since 'approval: done' was set, but this PR has failing CI tests.  Once the tests pass it will get moved to 'approval: ready to merge' automatically, alternatively please review and set the label manually.

Merged to master and 3.0.


CR branch

Finished looking through those files you specified yesterday in chat. LGTM, just a few comments/questions.


Apple logo

Please add support for Apple logo

@Davidvete Please refer to CONTRIBUTING.md for instructions for how contribute an icon to this library.


Add validation on prefix length

Prefix length can cause a name of a variable to become longer than is possible in the provider, find the length that is max for the prefix and set up validation in the DA to restrict to that length

Prefix removed with https://github.com/terraform-ibm-modules/terraform-ibm-account-infrastructure-base/pull/83 merge, closing

https://github.com/terraform-ibm-modules/terraform-ibm-account-infrastructure-base/pull/143 merged, closing


Search more from Lookup

Please Select if your Request is Either something new or an Enhancement

[x ] Enhancement of an existing Feature.
[ ] Request of a new feature.

Please select the area your request applys to. (Multiple selections are Possible)

[ ] Onboard. Initial vault setup and import/export pods
[ ] Create. Note creation, lookup, snippets and templates
[x ] Retrieve. Backlinks, references, graph view
[ ] Structure. Refactoring, multi-vault and schemas
[ ] Publish. Sharing your repo with the world
[ ] Misc (Choose this if your not sure)

Is your feature request related to a problem? Please describe
[suggestion] Search functionality. I might enter a concept as a heading or as a child note, this is a constant tension. If I try to find it, I have to search twice, in the Lookup and if not found, in the vscode search, possibly in vain as I might not have entered this concept at all. It would be better and faster if Lookup could trigger the full-text search, for example if the search term is not found, or upon user request via a key press, etc.
Describe the solution you'd like
Not 100% sure. This is a possible sequence:

find in file names
find in headings
find in tags
find in full-text workspace-wide search

as steps, or as a categorised list in results?
Describe alternatives you've considered
One can use vscode search, which is very good but requires searching twice.
Additional context
https://discord.com/channels/717965437182410783/739186036495876126/879372292587651122
https://discord.com/channels/717965437182410783/735365126227493004/919513734832074803

I would like to see the proposed enhancements, especially with note lookup also matching within note descriptions and tags!
I'd also like to add extra considerations for enhancing search.
vscode full text search has limitations

One can use vscode search, which is very good but requires searching twice.

I think the Lookup feature should go as far as searching front-matter and tags, but perhaps full-text is too much? Unless it was implied full-text only for the markdown note files dendron recognizes / processes and not all text in the workspace.
Automatically doing vscode full-text search could run into some performance issues if large attachments/assets happen to be included in the workspace, and although this should be negligible in most cases, probably not a good default to use/trigger from the lookup feature. vscode did work a few years ago to multi-process search, but it's not an indexed search and probably the reason it doesn't support fuzzy logic with the explorer full text file search.
fuzzy search
I'd like to add that if dendron supported fuzzy logic for searching notes, that would be great. While vscode file search can apply regular expressions, using re to implement NOT, AND and OR logic is not an intuitive task. However, indexing the note content is a non-trivial task. Technically, using file timestamps and a cache could help maintaining an index and avoid reindexing every time a workspace was opened.
I note that Zettlr does implement search a lot better than Dendron. See: https://docs.zettlr.com/en/core/search/. It supports search queries much like a search engine would allowing AND (implicit, space between keywords), OR (|), NOT (!) and exact match ("" quoted) operators. This is obviously more intuitive for most. I don't think it goes as far as doing proper stemming (https://en.wikipedia.org/wiki/Stemming) however. But still better than what Dendron and vscode offers.
As a powerful workaround to most note apps having limited full-text search features, I use a desktop search engine instead. E.g. recoll https://www.lesbonscomptes.com/recoll/ might not have the most modern UI, but it's got a powerful query language, similar to what you'd find with search engines, including Elasticsearch. The search syntax https://www.lesbonscomptes.com/recoll/usermanual/webhelp/docs/RCL.SEARCH.LANG.SYNTAX.html and built on top of Xapian (https://xapian.org/features), it can do extra things like:

Stemming of search terms
Synonyms
Phrase and proximity searching
Faceted search

While it's probably not a good idea to try replicate full desktop search engine features in dendron, perhaps dendron can try do at least some basic in-memroy fuzzy search match logic for note lookups.

Linking a related issue:

https://github.com/dendronhq/dendron/issues/394


(Broken) HTTP/1.0 server workaround for 'Connection: closed'?

Long story short
I need to communicate with a (broken|nonstandard) HTTP/1.0 embedded in a digital IP camera. This camera has a GET method and returns (text)lines of events, when they happen. The connection is supposed to be held open for hours and despite the server returning something like
HTTP/1.0 200 OK
Connection: closed
Content-type: text/plain; charset=utf-8
Last-modified: Thu, 01 Jan 1970 00:00:00 GMT

alarmimage=066267.3;timestamp=1486142859.602;date=2017-02-03;time=18:27:39;events=ima_51911cc4;eventlist="name=IMA 3,eventid=ima_51911cc4,type=IMA,sensortype=asen";alarmcounter=24704

the client is not supposed to close the connection here as more event lines will come eventually.
Expected behaviour
I would have hoped that using the aiohttp streaming API the connection would not get closed
Actual behaviour
aiohttp closes the connection, I guess the particular passage is in protocol.py:
        # read headers
        headers, raw_headers, close, compression = self.parse_headers(lines)

        if close is None:
            close = version <= HttpVersion10

Am I on the right track thinking it is aiohttp closing the connection? If so, how could I work around this
behavior?
Thanks!

client should work already in your case
https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/protocol.py#L318

It doesn't, however upon reconsidering I think it does not make sense to use aiohttp in such a scenario or tweak it to support broken HTTP servers. I ended up directly talking to the server via asyncio, formulating my "HTTP" request in place. Works for me.


fix(logging): improve python core's debug logging

As I'm looking at the changes and the README. Shouldn't we update the logging section of the README? Especially the code below. Now it's enough to enable the debug logging before the base service is created, right?
### Low level request and response dump
To get low level information of the requests/ responses:

from http.client import HTTPConnection
HTTPConnection.debuglevel = 1

As I'm looking at the changes and the README. Shouldn't we update the logging section of the README? Especially the code below. Now it's enough to enable the debug logging before the base service is created, right?
### Low level request and response dump
To get low level information of the requests/ responses:

from http.client import HTTPConnection
HTTPConnection.debuglevel = 1

Yes, I'm planning on doing that prior to merging.

@dpopp07 @pyrooka I think I can say that this PR is finally finished! :joy:  In my more recent changes, I've simplified a bit how testcases can enable debug logging, and I also updated the README.md to have more up-to-date info about logging.

Looks pretty similar so maybe it's good to go already but does this have any implications for how the SDKs do logging, as documented here?

Looks pretty similar so maybe it's good to go already but does this have any implications for how the SDKs do logging, as documented here?

Yes, I have a to-do item to go through the logging-related info in the common sdk repo and make adjustments as needed after all the logging-related changes in the cores.

:tada: This PR is included in version 3.21.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Purchasing payment

This PR completes the purchasing functionality and closes #422.

when a purchase has been made the payment (pending) section is shown.
a QR code is shown that can be used to pay for the purchase.
the payment address and amount can be copied.
the pay from wallet button shows a tooltip with the current wallet balance and a warning if the user doesn't have enough funds in their wallet.
if the user's wallet balance changes while the tooltip is open, it will re-render.
the pay with alt coins button will create a window for the shapeshift functionality (automatically populated by shapeshift).
when a purchase has been paid for, either with the internal wallet or an external source, the purchase complete section will be shown.
a link is shown that opens the transaction view and the transaction detail for that purchase.
a text area is present that can be used to send a chat message to the vendor.

Will start reviewing this PR soon!

I find this to be a strange experience @jjeffryes. All of @cpacia set moderators no longer appear to be offering moderator services, but it's still telling me to select a moderator.
If there a way to detect this scenario and eliminate the "Moderator payment" checkbox when necessary? The average person may not know how to move forward at this point.

@jjeffryes here's a few issues I found that will require a bit of clean up:

The vertical spacing between sections should be gutterV instead of gutterVMd

I noticed the coupon input doesn't always display. Any reason why?

Radio buttons would definitely be more intuitive for selecting a moderator instead of the check action. Is this change on hold?

Suggestion for a copy change on this error modal "You can't purchase your own listing" might be a better title.

Can you add a tx5 class to js-receipt? It feels a bit bulky as is.

Similar to above, can you make the values right aligned within js-receipt? Should look like this:

Clicking the Close icon should take you to the store view and clicking Return to listing should take you back to the listing detail overlay.

Buttons should have clrSh2

A few things in this copy block... The link should be green and point to https://www.openbazaar.org/bitcoin/. Also some of the copy is off since we're not doing timed releases. Please update copy to: "Your payment will be held securely in escrow. If any issues arise, you’ll have the ability to open a dispute." Also, that copy should only display if/when the transaction type is multisig.

I'm going to continue testing a bit more to see if anything else comes up.

I'm running into this error pretty often now after clearing my database. What does it mean?

Also, I can't highlight the Pay amount on this screen

@morebrownies I think if you get the signature error it's because the vendor needs to update his listing. It's old code.

@jjeffryes Could also merge in master and fix the conflict?

The Purchasing modal bombs when one of the shipping options is a local pickup:

There are no services in that case... so maybe that's causing the issue..?

I noticed the coupon input doesn't always display. Any reason why?

@morebrownies if the listing doesn't have coupons, the coupon input isn't shown. That was in the requirements.

Radio buttons would definitely be more intuitive for selecting a moderator instead of the check action. Is this change on hold?

Yes. This component is also used in settings/store, which doesn't have a radio button. This is a compromise between the designs of the two, since they don't work the same.

@morebrownies and @rmisio this PR has been updated.
@cpacia I'm going to make the updates to the notification format a separate PR.

Awesome, looking forward to testing this out. I'm planning to spend some time on it tomorrow morning.

There seems to be a bug here. The free shipping option is checked by default, but when I try to proceed to pay, I get hit with the "Please choose a valid shipping option" error.

The issue above happens when trying to purcash this item ob://vintagefashion/store/zebra-dress

@morebrownies that issue should be fixed now.

Alright just had a chance to play around with this some more and it's looking very good. Nice work @jjeffryes
My 2 pieces of feedback at this point are:

We should consider always having the coupon input visible, regardless if the vendor has active coupons or not. As a buyer, I may have a code and not know that is has expired, but if the coupon input is hidden, there's no way for me to test the code. It would be a better experience to let the user enter a invalid coupon and receive a error message than to leave them confused.

I brought this up above and I'm assuming we'll revisit it after launch, but wanted to call it out one more time in case it was missed. The checkbox experience for moderators is strange. Ideally we would use native radio buttons here instead of the custom check button. It would be best to have the radio buttons on the left side of the avatar too, as pictured here:

Other than that, looking very good. We'll definitely want to have the community hammer on this and see if they get caught up anywhere too, but overall feels like a smooth checkout flow; much much better than 1.0.

I spoke too soon, just ran into this issue:

I can't finish purchasing because no mods are online, even when the Moderated Payment option is unchecked.

Ideally we would use native radio buttons here instead of the custom check button.

@jjeffryes We could circle back to this after launch... could that mod view be abstracted to where it takes an option on whether to use a radio button or a checkbox? If it's a radio, there could be a hidden radio button that you could select when you don't want the mod selected. I'm assuming some view already handles the logic to deselect all other mods when one is selected, so I would think that could be abstracted to handle the radio button case.

Sorry, one other issue I noticed...
Hitting "BUY NOW" on a listing with a price of 0 does not do anything. We should have a error message trigger when a user clicks "BUY NOW" on a free listing.

We should have a error message trigger when a user clicks "BUY NOW" on a free listing.

Are we allowing free listings?

I believe the free listings are coming from the mobile app.

I believe the free listings are coming from the mobile app.

If we've made the decision that we can't allow free listings because it would have to change the server architecture (since there's not BTC transaction) and we're already not allowing coupons to take a listing at or below zero.... shouldn't we follow through and have the server (and then clients) prevent a 0 amount listing from being saved?
@jjeffryes ^

No, free listings shouldn't be allowed by the server, since it's not possible for them to ever have a successful transaction.
Weirdly, the purchase API allows you to make a purchase of a zero cost item.
I'll put in a warning so you can't buy something that costs zero, but it should be fixed on the server so it's not possible (there could still be rogue servers that allow it).

https://github.com/OpenBazaar/openbazaar-go/issues/546

@morebrownies that bug with the moderators is fixed, and I put in the radio button. I kept the view button, because the user still needs to be able to look at the details of a moderator (clicking anywhere but the radio button will show the details, but I think there should be a visual prompt for it).
The listing price will update according to the variant selected now, and an error message will be shown if the total price is zero. This would allow a zero price listing with a variant that adds a price, which is kind of weird but would still be a valid transaction.
There is also code in the purchase view that makes sure the price is positive, just in case somehow they get to that screen with a zero price listing, or a negative shipping or variant price is somehow applied.
And the coupon entry is always visible now.

Hey @jjeffryes I noticed local pickup options have issues. I have one selected but can not continue to check out.

@morebrownies can you give me the link to that listing? I can't reproduce the issue.

ob://QmbG9sv6Veox66CoretT614XGmXqwXPzdUFfitDQLMJWtM/store/milly-beamer

@morebrownies it works fine if your address is in albania. :)
There was a bug that was showing shipping options you can't use. That's fixed now.

@morebrownies this is just waiting for your final thumbs up before we merge it.

Ok great. I'll do a final pass after I wrap up Tor designs. Shouldn't be too long.

Looking good to me. All the issues I posted above seem to be resolved. 👍


CHORE - Remove Trailing Spaces

Remove Trailing Spaces:
Why:
Saves space
Reduces silly conflicts

Coverage remained the same at 97.557% when pulling e589625db179b3a30c708cb53b18421a8441fb06 on clickthisnick:chore-remove-trailing-spaces into ad8ab1d7a5284407efb83d0d7a8b32eb6fbd93c4 on gotwarlost:master.


FFI

We need a better way to generate FFI (foreign function interface) language bindings.  We need to gather all the available ways to generate those bindings so we can make a choice that works with our desired target platforms.

FFI Genenerator
FFI Generator Docs
Swift (iOS)
Kotlin (Android)
Native (macOS, iOS, tvOS, watchOS, Linux, Windows, Android NDK)
C# (Windows)
C
C++
Result

Manual

✅
✅

✅

Custom

swift-bridge
swift-bridge Book
✅

RustResult

UniFFI
UniFFI user guide
✅
✅
✅ uniffi-kotlin-multiplatform-bindings
✅ uniffi-bindgen-cs

throw

Diplomat
The Diplomat Book

✅
✅
?

Interoptopus
Docs

✅
✅

?

Future issue for @KronicDeth

At 2023-08-17 meeting the team decided to do an exploration with UniFFI to see if it meets our needs for both Kotlin/Android and Swift/iOS.


Spec patterns did not match any files.

$ node -v  v4.3.0
$ protractor --version  Version 3.0.0
Gruntfile.js snippits
protractor_coverage: {
                options: {
                    keepAlive: true, // If false, the grunt process stops when the test fails.
                    noColor: false, // If true, protractor will not use colors in its output.
                    coverageDir: '<%= <app name>.instrumentedE2E %>'
                 },
                local: {
                    options: {
                        configFile: "./test.old/protractor/conf.js"
                    }
                }
            },

Task runner
   return grunt.task.run([
                       'env:all',            // Set local config variables
                       'env:test',           // Set NODE_ENV=test
                       'express:dev',        // Start the server in dev mode
                        'clean:coverageE2E',
                        'copy:coverageE2E',
                        'instrument',
                        'protractor_coverage',
                        // 'protractor',
                        'makeReport'
                    ]);

Error
/Users/<name>/<app name>/node_modules/protractor/node_modules/q/q.js:126
                    throw e;
                    ^

Error: Spec patterns did not match any files.
    at Runner.run (/Users/<name>/<app name>/node_modules/protractor/lib/runner.js:249:11)
    at TaskRunner.run (/Users/<name>/<app name>/node_modules/protractor/lib/taskRunner.js:123:19)
    at createNextTaskRunner (/Users/<name>/<app name>/node_modules/protractor/lib/launcher.js:223:20)
    at /Users/<name>/<app name>/node_modules/protractor/lib/launcher.js:246:7
    at _fulfilled (/Users/<name>/<app name>/node_modules/protractor/node_modules/q/q.js:797:54)
    at self.promiseDispatch.done (/Users/<name>/<app name>/node_modules/protractor/node_modules/q/q.js:826:30)
    at Promise.promise.promiseDispatch (/Users/<name>/<app name>/node_modules/protractor/node_modules/q/q.js:759:13)
    at /Users/<name>/<app name>/node_modules/protractor/node_modules/q/q.js:573:44
    at flush (/Users/<name>/<app name>/node_modules/protractor/node_modules/q/q.js:108:17)
    at nextTickCallbackWith0Args (node.js:419:9)

Config and Spec file
new.spec.js
describe('it should laod the page',function () {
    it('should launch the page',function () {
        var loginURL = '<ourInternalURL>:3001/#/';

        browser.get(loginURL);
        browser.sleep(10000);
    });
});

conf.js
exports.config = {
    multiCapabilities: [{
        'browserName': 'chrome'
    }],
    //seleniumAddress: 'http://localhost:4444/wd/hub',
     specs: ['./new.spec.js'] // TODO comment out for now due to the new login page, as of 1/8/16
};

When i swap our protractor_coverage with protractor in the task runner the tests execute fine but they are not running against the instrumented code so I dont get the reports.
I have tried this with other conf and spec files that have been pulled from a repo upon npm install.  The spec and config files here are in the actual app hierarchy.  I'm not sure what the issue is.  Let me know if you need anything else.

specs: ['./new.spec.js']
refactored to
specs['**/*.spec.js'] and everything works


DeferredStorage

I can revert the rename if we want, but I'd like the other changes since they make the storage a lot nicer to use.

Renames RestrictedStorage to DeferredStorage.
Removes Entry/EntityIndex in favor of just storing the index in the returned storage.
Also removes a clone from the mutably RestrictedStorage.

Should we have Deref/DerefMut access on the PairedStorage? It could be handy for cleaner code, but it would make the deferred access a bit more implicit which could lead to some more annoying maintenance bugs for people. (Accidentally move a mutable modification outside of a condition which then marks all components are modified).

This change is

Should we have Deref/DerefMut access on the PairedStorage?

No, I think it's better to remove it and make the access explicit.

Reviewed 5 of 8 files at r1.
Review status: 5 of 8 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

@torkleyy IIRC this is ready (just need to rebase)

Looks good to me. Some more reviews would be helpful, cc @Rhuagh @Xaeroxe

Reviewed 1 of 8 files at r1, 5 of 6 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

src/storage/restrict.rs, line 32 at r2 (raw file):

/// Similar to a `MaskedStorage` and a `Storage` combined, but restricts usage
/// to only getting and modifying the components. That means nothing that would

Err I think you accidentally removed some lines

Comments from Reviewable

Reviewed 2 of 8 files at r1, 5 of 6 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

Comments from Reviewable

Review status: 0 of 7 files reviewed at latest revision, 1 unresolved discussion.

src/storage/restrict.rs, line 32 at r2 (raw file):
Previously, torkleyy (Thomas Schaller) wrote…
Err I think you accidentally removed some lines

Done.

Comments from Reviewable

Reviewed 7 of 8 files at r3, 1 of 1 files at r4.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

bors r+


Delete main.js

Delete now

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: jeepdude1980 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Can anyone help me reset my Java to factory I was hacked
On Sat, Feb 8, 2020, 1:04 AM Richie Bendall<EMAIL_ADDRESS>wrote:

DO NOT MERGE
This guy is sending pull requests to a bunch of repositories with requests
to remove critical files.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Azure/login/pull/12?email_source=notifications&email_token=AOM7YOINJQKDU6O2IUB4W63RBZDQLA5CNFSM4KQU7LR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFKRVI#issuecomment-583706837,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOM7YOLJBANRXD2YSGITTKTRBZDQLANCNFSM4KQU7LRQ
.

no


Checker Do Actual Computation (lang: SML) (bug?)

Hi,
I'm using Syntastic for Standard ML. The syntax checker (automatically upon opening the source file) seems to be evaluating my program.
I realize this because I have an expression evaluating Ackermann function on (4, 1). Opening the file takes really long time. When I disable Syntastic, opening the file becomes instant fast.
Is there a way to get around this? Or is this something to do with the checker itself?
Thank you!

In order to see what syntastic does you can enable debugging (cf. :h syntastic-debug).
In order to turn off automatic checking of your files you can set sml as a passive filetype (cf. :h 'syntastic_mode_map').
In order to check files in background you can use an asynchronous checker instead of syntastic, such as ale.


Exposed MigrationsExtension.

#483

:white_check_mark: Build ExtendedXmlSerializer <IP_ADDRESS>-hnoqgddx completed (commit https://github.com/ExtendedXmlSerializer/home/commit/4fe29ebd13 by @Mike-E-angelo)
Install it by running the following command in Package Manager Console:
Install-Package ExtendedXmlSerializer -Version <IP_ADDRESS>-hnoqgddx -Source https://ci.appveyor.com/nuget/extendedxmlserializer-preview

Problems with the above command?  You may be running into this known issue here:
https://github.com/NuGet/Home/issues/7189

:white_check_mark: Build ExtendedXmlSerializer <IP_ADDRESS>-gcoqndih completed (commit https://github.com/ExtendedXmlSerializer/home/commit/aec41bd5be by @Mike-E-angelo)
Install it by running the following command in Package Manager Console:
Install-Package ExtendedXmlSerializer -Version <IP_ADDRESS>-gcoqndih -Source https://ci.appveyor.com/nuget/extendedxmlserializer-preview

Problems with the above command?  You may be running into this known issue here:
https://github.com/NuGet/Home/issues/7189

:white_check_mark: Build ExtendedXmlSerializer <IP_ADDRESS>-heqhxqkc completed (commit https://github.com/ExtendedXmlSerializer/home/commit/f999178170 by @Mike-E-angelo)
Install it by running the following command in Package Manager Console:
Install-Package ExtendedXmlSerializer -Version <IP_ADDRESS>-heqhxqkc -Source https://ci.appveyor.com/nuget/extendedxmlserializer-preview

Problems with the above command?  You may be running into this known issue here:
https://github.com/NuGet/Home/issues/7189


Perplexity API doesn't work

[x] This is actually a bug report.
[ ] I am not getting good LLM Results
[ ] I have tried asking for help in the community on discord or discussions and have not received a response.
[ ] I have tried searching the documentation and have not found an answer.

What Model are you using?

[ ] gpt-3.5-turbo
[ ] gpt-4-turbo
[ ] gpt-4
[x] Other (please specify)
pplx-70b-chat

Describe the bug
Using the Perplexity API throws a 400:
openai.BadRequestError: Error code: 400 - {'error': {'message': '["At body -> response_format -> ResponseFormatText -> type: Input should be \'text\'", "At body -> response_format -> ResponseFormatJSONSchema -> type: Input should be \'json_schema\'", "At body -> response_format -> ResponseFormatJSONSchema -> json_schema: Field required", "At body -> response_format -> ResponseFormatRegex -> type: Input should be \'regex\'", "At body -> response_format -> ResponseFormatRegex -> regex: Field required"]', 'type': 'bad_request', 'code': 400}}

To Reproduce
Run the example script from the repo: https://github.com/jxnl/instructor/blob/main/examples/open_source_examples/perplexity.py
Expected behavior
The script should run successfully.

This is a known issue, perplexity does not do structured outputs well


bpfilter errors flooding the console

Kairos version:
Alpine version
CPU architecture, OS, and Version:

Describe the bug
The Alpine image is functional, however messages are flooding in the console so it's impossible to use it:

To Reproduce

Expected behavior

Logs

Additional context

dup of #277


request for buf_execute() function

In latest version of vim, we have a function called win_execute(), this is usful to run command without switch windows. and it is better to has a function called buf_execute(), just same as win_execute(). It should be able to run command on specific buffer without switch to that one.

When a buffer is not in a window, many commands will fail.  E.g. there is no cursor position.  The only way operations can be performed on a buffer properly is to have it in a window.  Some commands might work without a window, but this is unpredictable.  Internally there are a few tricks to operate on a buffer, but many times a problem was uncovered.  Autocommands also make this difficult, since they can do anything (often unintentionally).

@brammool Thank you for your reply. What I need to achieve now is to save the specified buffer without switching to that buffer. :bufdo commands switches buffers.

What I need to achieve now is to save the specified buffer without switching to that buffer.

If you don't care about being specific, you can execute :wall,  to save all changed buffers.
If you do want to be specific, then indeed we would need a new function; bufwrite() or buf_execute() for example.
In the meantime, it's a hack, but maybe you could display the buffer in a hidden popup, then call win_execute():
vim9script
var buf: number = bufadd('/tmp/file')
bufload(buf)
appendbufline(buf, 1, ['some', 'text'])
var id: number = popup_create(buf, {hidden: true})
win_execute(id, 'noautocmd write')
popup_close(id)

Thanks, hidden popup is a nice idea.

Speaking of hacks,
call getbufline(bufno, 1, "$")->writefile(bufname(bufno))
call setbufvar(bufno, "&modified", v:false)

@matveyt thanks, it works well for me, but I think you should use call setbufvar(bufno, "&modified", 0), when use v:false, I get E521: Number required: &modified = 'false'

Maybe this is a bug, we should be able to use v:false to set &modified.

Related issues:

#7603
#7615

They were fixed about a year ago in patches 8.2.2284, 8.2.2285 and 8.2.2297.

How do you define "without switching to that buffer"?

Speaking of hacks,

I wouldn't call this a hack

let's close this.


On dismiss closure feature request

Hey
This pod is so so good and such a pleasure to use. Thanks for making and sharing it!
I have a feature request. Would it be possible to add an on dismiss callback closure which would be triggered when the formsheet style gets dismissed.
Thanks!

Hi, @heliumdice thanks for the wonderful suggestion.
This has been added in latest version 4.0
you can use it like below:
// get on dismiss event with selected items
selectionMenu.onDismiss = { selectedItems in
   self.selectedDataArray = selectedItems
}

optionally you can comment the line - for updating array on each selection event and you'll get final selected items on menu dismissal.
// show selected items
selectionMenu.setSelectedItems(items: selectedDataArray) { (text, selected, selectedItems) in
  //self.selectedDataArray = selectedItems
}

Oh wow that is amazing! So quick.
I implemented it today and it works a treat!
Thanks you!


Only recurse specific dirs

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
http://build-eu-00.elasticsearch.org//job/Puppet_Elasticsearch_PR/40/
Test FAILed.

test this please

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
http://build-eu-00.elasticsearch.org//job/Puppet_Elasticsearch_PR/41/
Test PASSed.


[Style Transfer][iOS] Added a Style Transfer example for iOS

Summary
To learn more about TF-Lite and help others start learning, I put together a basic example for how to do Style Transfer on iOS. It is inspired by the corresponding Android example and uses the iOS Image Classifier example as a base.
Apologies for the large diff. Since it's my first time using TF-Lite, I wrote it first by incrementally by modifying the ImageClassifier example and then copied to to a new folder once it was in its current state.
Since it's based on the ImageClassifier example, which does real-time inference of a video feed, this example performs style transfer on a video feed as well. However, it's possible to modify it to be closer to the Android UI (the TF-Lite maintainers can decide), but I think it's pretty neat to see in real-time.
Notes to maintainer

Let me know if I have to update the copyright messages at the top in any way

Next steps

Add options for the user to select a style: Done in #184
Let the user take a still picture: Done in #185

Video

Hey Ivan, thanks for the contribution! We need to sync internally on some of these proposals, so thanks for your patience until we get back to you (hopefully in the next few days). Cheers.

@jdduke Sounds good! Let me know if you want it to match the Android example exactly. It'd be a fun exercise for me. There's definitely merit in that as well and it'd be pretty easy to switch it over.

@ivanmkc Thanks for the contribution and sorry for the delayed reply.
We already implemented a style transfer iOS sample internally and it's due to release very soon. Due to  internal requirements, it's quite difficult to for us to accept a full external sample and adapt to our internal requirements. Therefore, we'd prefer to receive contribution to our existing samples rather than a new sample. I'm sorry for the limitation.
Your sample has nice feature that our internal sample don't have, including the ability to stream video. We'd love to see you extend our sample after it has launched.
Meanwhile, the TF Lite community has put together a repository collecting community-made samples. Would you be able to host the sample on your GitHub repo, and add it to the list?
https://github.com/margaretmz/awesome-tflite

@khanhlvg sounds good! It was a very educational exercise nonetheless and I'll go ahead and add to the community-made repo then.
Looking forward to the official style-transfer sample.

Thanks @ivanmkc ! I'll close this PR for now and looking forward to your contribution once our sample is out :)


Man_made Color representation

Today I realize that the breakwaters are now shown in blue and dotted from the previous white color; as been a man made structure my logic says that should be depicted with a different color that adjacent features like water, so keeping a different colors made sense, like wise for Dams and Weirs that are already blue two elements of different nature however related needs to be easilly spotted.
Could these be fixed? has been discussed somewhere?
Thanks

I changed the color to use the same dark grey that we use for rocky things (like quarries).

I don't think weirs even have a preset. I guess they can get custom styles as well.

Oh btw, there is a weir preset, but they are defined only for vertices and lines, not areas.
The dam preset is allowed on all geometry types.
This is following the guidance on the OSM wiki.


Improved "Possibly unhandled rejection" error handling for AngularJS 1.6+

Subject of the issue
This is more of an enhancement request than a bug report.
Since AngularJS 1.6 rejected promises that are not handled by a catch block results in an exception with the message "Possibly unhandled rejection".
If I use the $translate service to get a translation by a dynamic id it gets very difficult to track down the source of the problem if undefined is passed as translationId.
Your environment

version of angular-translate: 2.18.1
version of angular: 1.7.2
which browser and its version: Google Chrome 69 on Windows 10

Steps to reproduce
http://next.plnkr.co/edit/nEpRbSKThpDXq3Qs?open=lib%2Fscript.js&deferRun=1
See the exceptions in the console.
Expected behaviour
More informative error handling. Stack trace would be nice.
Actual behaviour
The translationId is passed to the angular's exceptionHandler.

undefined as a translation id (aka input argument) should be checked right in the service, before even requesting the internal stack.
So I think returning the rejection asap is far besser. Maybe you want to make the PR?

How does "returning the rejection" give more information about where the invalid argument was passed? Not questioning at all I am just not very familiar with the source code.

Well, if a function is returning a promise, it would be a surprise handling for exceptions also. That's why Promise.reject() exist.

@knalli It took a little over a year but I have submitted PR #1891. Not sure if it is what you meant but it should pass unit tests.

good request.. .closed w/o a solution

@xhlatky What exactly is the "rude way" here?

no valid explanation for me
Dňa št 12. 11. 2020, 16:08 Jan Philipp<EMAIL_ADDRESS>napísal(a):

@xhlatky https://github.com/xhlatky What exactly is the "rude way" here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/angular-translate/angular-translate/issues/1860#issuecomment-726135965,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD5CAMIBE7CJC2ZOCCCCCI3SPP253ANCNFSM4FY6H77Q
.


Error in Jenkins

ERROR: Credentials 'docker' is of type 'Username with password' where 'org.jenkinsci.plugins.plaincredentials.StringCredentials' was expected
Finished: FAILURE

Use Credential bind as "seprate username and password and use to mension the env pass as variable password.


Block: Add story selector for embedding single story

Feature Description

Splitting this out from #6894
We want to make the Single Story embed block more similar to the image block, allowing to choose between selecting an existing story or inserting from URL:
Mock:

The URL option would show the URL input field in a popup/tooltip like in the Image block.
The "Select Story" button would open the story selector we already have today, but only allow selecting 1 story (and not show a reorder option, obviously).
Alternatives Considered

Additional Context

Verified in QA


API server should limit the number of concurrent requests it processes

Related with issue #5865, the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.

+1 !

The good news/bad news on this is that there is a rate limiter in the API server (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/apiserver/handlers.go#L70), but the defaults for it seem too high: 10 qps average, 200 burst (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cmd/kube-apiserver/app/server.go).
We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.

I think that we need limits on the number of outstanding requests, not the number of QPS.

@quinton-hoole -> An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests.
@smarterclayton -> Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue #5864).

I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled.  Being able to timebox and classify requests differently would be good.
----- Original Message -----

@quinton-hoole -> An max outstanding would be useful. But it may force us to
have separate buckets for different types of requests, i.e. a small number
of long running requests causing API server to reject short requests.
@smarterclayton -> Heard that you might be looking into adding support for
flow control (or something like it?). We may need it soon to help avoid bugs
like runaway node status updates (see issue #5864).

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/5866#issuecomment-85711150

@smarterclayton - Ah, np, I probably misheard.

What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the client library?
I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages.
This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to #5864 it'll probably get worse.

The client library should have options for retry on client.Config which include the ability to disable the behavior.  Automatic retry is probably an "opt-out" behavior.
----- Original Message -----

What are our plans for handling this client side? Do we want clients to
handle "you're being throttled" error themselves, or do we plan to handle it
in the client library?
I'm asking because currently kubelet does not have a 'short' retry loop,
hence it tries to report NodeStatus every X (2) seconds, whether it succeed
or not. Neither we have a sidechannel for heartbeat messages.
This would mean, that if API server will be under heavy load, and with
not-negligible probability NodeStatus reporting will be throttled, we may
incorrectly mark a Node as unreachable. Currently it's not a huge problem,
because X << Y, where Y is the time we wait before we decide a Node is
unreachable. But because of the effort somewhere close to #5864 it'll
probably get worse.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/5866#issuecomment-90459186

@lavalamp says there are people working on rate limiting client connections to API server.

I have in the past adjusted the rate limiter to support the creation of large clusters. I also added the Retry-After logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.

I'm closing this as fixed, given the original subject.  If we want to expand client-side support for retry, please open a dedicated issue.  (Also note that QPS based throttling was added to the client recently too)


7x/field resolver performance

This brings some performance enhancements to the way we resolve Field, Fields and PropertyName when these are instantiated using an Expression. E.g if you do q.Term<Project>(p=>p.Name, "value")
We were visiting the expression multiple times

To create a comparison value we used a specialized visitor
This visitor calls tostring which in turns calls the BCL tostring visitor
Then we did some regex replace to create a sane key
Visit expression and check if there are compilable parts and if so make sure we do not cache the expression
Finally resolve the expression using the connection settings using another visitor

This PR
Combines the comparison vistor/tostring visitor/caching visitor into one. We no longer rely on Expression.ToString() and since we only need to deal with a certain type of expression our tostring performance better then the BCL one.
Then if its we pass it off the FieldExpressionVisitor if not in local cache already. While writing this I wonder if FieldExpressionVisitor could reuse the stack from our new ToStringVisitor. Will follow up in a comment.

Don't see any real benefit and it adds quite some complexity so resetting locally.


[BUG] Network Error trying to load video dataset in the App

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
Python version (python --version): 3.8.5
FiftyOne version (fiftyone --version): 19.1
FiftyOne installed from (pip or source): pip

Commands to reproduce
As thoroughly as possible, please provide the Python and/or shell commands used to encounter the issue. Application steps can be described in the next section.
fiftyone app view --videos-dir /data1/videos

Describe the problem
Work fine for images dir, (Dataset.from_images_dir), but fails for videos. No showing in website browser.  Local Network, No Internet connection.
Code to reproduce issue
Also start the FiftyOne App using Python:
import fiftyone as fo

dataset = fo.Dataset.from_videos_dir("/data1/videos")
session = fo.launch_app(dataset, remote=True)
session.wait()

Same with starts command line, tell Network Error.
Other info/logs

What areas of FiftyOne does this bug affect?

[x] App: FiftyOne application issue
[ ] Core: Core Python library issue
[ ] Server: FiftyOne server issue

Willingness to contribute
The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

[] Yes. I can contribute a fix for this bug independently
[ ] Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community
[* ] No. I cannot contribute a bug fix at this time

I was able to reproduce this error by uninstalling ffmpeg, so I think this is indeed the issue.

@benjaminpkane I believe the intention was for the user to see a nice error:
https://github.com/voxel51/fiftyone/blob/dfb81b6fffb8ef9bb36bcb0d534f357e664ccfc8/fiftyone/server/metadata.py#L160-L163
when they don't have ffmpeg installed, but this
https://github.com/voxel51/fiftyone/blob/dfb81b6fffb8ef9bb36bcb0d534f357e664ccfc8/fiftyone/server/metadata.py#L87-L89
doesn't seem to be getting it into the App's error handler (anymore?)
Can we get the intended error message to show in the App?

Hmm I'm not able to reproduce. This works for me, for example:
fiftyone zoo datasets download quickstart-video

VIDEOS_DIR="$(fiftyone zoo datasets find quickstart-video)/data"
fiftyone app view --videos-dir ~/fiftyone/quickstart-video/data

Do you have ffmpeg installed? That is required in order to work with video datasets: https://docs.voxel51.com/getting_started/install.html#installing-fiftyone

Thanks for your kindly reply. Unfortunately, I can not download that QuickStart-video directly for server is deployed in a local network.
What interesting is, the image dataset (load_images_dir) works perfect, so I guess there is no connection error.
I will try to download quick-start video data in an Internet access PC, and then copy part of video into a temporary directory to see whether the problems is cased by video format.

@benjaminpkane I believe the intention was for the user to see a nice error:
https://github.com/voxel51/fiftyone/blob/dfb81b6fffb8ef9bb36bcb0d534f357e664ccfc8/fiftyone/server/metadata.py#L160-L163
when they don't have ffmpeg installed, but this
https://github.com/voxel51/fiftyone/blob/dfb81b6fffb8ef9bb36bcb0d534f357e664ccfc8/fiftyone/server/metadata.py#L87-L89
doesn't seem to be getting it into the App's error handler (anymore?)
Can we get the intended error message to show in the App?

Thanks for your reply. I start the fiftyone in a remote server, in methods of python script and command line, both got the same errors. I am a new for fiftyone family, and I installed fiftyone yesterday, version is 19.1.
FFmpeg is installed, I checked that several times.
The video is also process using fiftyone.utility.videos.reencode_video func provided by fiftyone lib. All video could be opened correctly using a media player.
Strongly, there is no intended error message showing in server, but fails in PC’s website browser. whether there is a way to print the error message? Just like verbose mode?

I doubt this is not error about video format (of course no connection problem, works fine for image folders). I start the process (videos-dir) using an empty dir or putting quick-start video, the errors stills appears.

Problems solved. Updater the website browser in PC, and fiftyone on server updated to 0.19.1rc1 version. Works fine!


Move most of TerminalApp's runtime Xaml to a .xaml file and class

Summary of the Pull Request
This pull request migrates a lot of the manual Xaml construction from TerminalApp::App into a .xaml file. The benefits in doing this are myriad:

We can now use the live xaml designer to edit Terminal's layout!
It performs way better because we don't need a WinRT call for every object's every property
It is the groundwork for supporting multiple windows
It can be themed with theme resources without jumping through terrible, terrible hoops
The Xaml compiler produces all of the members for all of the named xaml objects all on its own!

It is, however, incomplete. I have only moved the xaml, I have not moved any of the logic. This is groundwork, and necessary for @cinnamon-msft to keep working on the UI. Followup issue #1878.
References
Mentioned in #929.
PR Checklist

[ ] Closes #xxx
[x] I've discussed this with core contributors already.

Validation Steps Performed

Manual run with custom themes, settings, toggling "always show tabs" and "tabs in titlebar."

AcrylicBrush supported?

This does not suddenly remove the requirement that we are hosted in a win32 application in a xaml island, and it doesn't change whether the acrylic brush is supported. That is to say: it already works today, it could have been done with code, but if you are in showTabsInTitlebar mode you're going to get an ugly black box in the middle of your acrylic.
The acrylic brush is already being used for the background of the TerminalControl, anyway.

Illustratively:

If you ask why there's a black box, I'll have to refer you to #1870 #1524 #564 #1395 #1539 #929 because the xaml island is an input site we don't have message loop control over, we can't get the titlebar hit test out of it to allow dragging clicks to originate inside a xaml region.


[$1000]  uploading EPS extension file throws an error

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open any chat
Click on :heavy_plus_sign: icon
Click on Add attachment
Select .EPS extension file and send it

Expected Result:
The app should send the file or throw "This filetype is not allowed" pop-up.
Actual Result:
Gives an error saying "Invalid image supplied"
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.53-1
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/43996225/c4a557de-6ed6-4915-b4b0-de2a2a34a42d

Expensify/Expensify Issue URL:
Issue reported by: @jo-ui
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1690751127702569
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01441ec377d38bb1cc
Upwork Job ID:<PHONE_NUMBER>932484608
Last Price Increase: 2023-08-21

we are not validating dropped file extension. if there are some restricted extensions we should add a condition to validate.
 const {fileExtension} = FileUtils.splitExtensionFromFileName(lodashGet(file, 'name', ''));
    if (_.contains(CONST.API_ATTACHMENT_VALIDATIONS.UNALLOWED_EXTENSIONS, fileExtension.toLowerCase())) {
        setIsAttachmentInvalid(true);
            setAttachmentInvalidReasonTitle('attachmentPicker.attachmentTooSmall');
            setAttachmentInvalidReason('attachmentPicker.sizeNotMet');
            return false;
        return false;
    }

I think It is BE bug. The error is threw from BE side

@strepanier03 any updates on this issue?

@strepanier03 It has been a week since this issue was posted, but there has been no response.

I was on sabbatical and am just getting back from it now. I'm working on this today and will have an update soon.

I was able to recreate this behavior using a random .eps file on my computer as well as the one shared in the Slack bug report thread.

@burczu - If you think this is an internal issue please let me know and I can switch to an Internal label.

I can upload the files without any issue on staging.

Similar issue was noted here - https://github.com/Expensify/App/issues/22306#issuecomment-1630209033
I tried to create slack conversation for this, but didn't get any response - https://expensify.slack.com/archives/C01GTK53T8Q/p1690348144733229
Proposal
Please re-state the problem that we are trying to solve in this issue.
uploading EPS extension file throws an error
What is the root cause of that problem?
Some files' type is empty when checking in javascript, and backend is resulting in error for those files for example: .c files. For above mentioned .eps file it is working normally for me but I am assuming sometimes it might fit above case.

if type is empty the backend also rejects this file, and sends an error.

What changes do you think we should make in order to solve the problem?
So there is no point of sending files where type is empty. We can add additional condition !lodashGet(_file, 'type') here to check if type is empty:
https://github.com/Expensify/App/blob/acf9cede7f0f5f4c2ebd89cc48f012de6ae81b0f/src/components/AttachmentModal.js#L186
This way, we can any files which backend would respond with an error.
What alternative solutions did you explore? (Optional)

@strepanier03 The Invalid image supplied message that is added after the Unexpeceted error... message comes from the BE side, so I think it means that BE has some issue processing files with this extension. That's why I believe we should we should make it internal and at least check what's happening with these files on the BE side.

@burczu I agree that error comes from BE.
But IMO allowing uploads for all file type to backend could potentially expose the application to security vulnerabilities. What do you think?

@ishpaul777 from what I see the app allows to upload any type of file - if it's not an image or pdf then you can just download the file (there is no preview in the report) and I guess it is done this way by design

Platforms like Slack, Gmail and Google Chat have implemented restrictions on certain types of file uploads (references: Slack policy, Gmail and Google Chat policy). Considering this, it might be prudent for us to implement similar restrictions on the types of files users can upload. I recommend that we discuss and consider implementing a file type validation mechanism that only allows safe and necessary file types for upload.
Allowing all file types, including potentially harmful ones like executable files (e.g., ".exe") exposes vulnerabilities for users as well as backend, However, if this is intentionally part of the design, I recommend that we document the rationale behind it, along with the assessed risks.

@burczu - reading the response from @ishpaul777 here, do you want to raise this for discussion at all or should I switch the labels now?

@strepanier03 - I think for this issue we should just switch the labels
@ishpaul777 - I can hear you and it's reasonable what you wrote, but I think it sounds more like a feature request so I believe you could raise another issue to discuss it - what do you think?

Can you chime in slack thread https://expensify.slack.com/archives/C01GTK53T8Q/p1692691281183839

Nothing yet on the thread but it's Friday so we might have some stragglers come in Monday. Will bump then.

No new movement on the thread so I posted in another internal channel trying to get more opinions on it. We could also allow the internal engineer to move this forward more when it gets picked up as well.

I'm really struggling to get opinions weighing in here.

Discussion is ongoing on the thread, will continue to follow along.

@ishpaul777 @burczu - Based on the discussion in the thread, do we have a path forward?

there is no discussion around the https://github.com/Expensify/App/issues/24497#issuecomment-1688006744 in the thread, i believe the unallowed extensions are removed  intentionally,
To move forward with This particular issue(not able to upload .eps file ) i think you should post issue on slack internally, to get more eyes on this from internal engineers
@strepanier03

@strepanier03 I agree with @ishpaul777 - this thread on slack was about his concerns on adding additional validation for file types on the frontend side. It separate thing and this issue should be still fixed on the backend side - that was the reason to change the label to internal.

Just to inform: I'll be OOO on Thursday and Friday (7-8 September).

@ishpaul777 and @burczu - I will bump the internal thread I started for this previously. Thanks for the heads up about your OoO as well.

I bumped this in Slack and @Gonals had worked on this. It's on our radar for now.

Issue not reproducible during KI retests. (First week)

@mvtglobally - Thank you!

Still waiting for internal ownership, I'll bump again internally this week.

Hey @strepanier03 I had a second to look into this - I'm not seeing the error, on staging or production. Are you still able to reproduce? A sample .eps file I used uploads and downloads fine.
Not sure what's changed since the issue was opened but we might be all set now.

@dangrous Try it with other .eps files.
EPS files.zip

@dangrous - I'll give it another shot.

@dangrous - Just tested again with the three .eps files that @jo-ui shared here. All three repro'd the reported behavior.

Ah yep, tried some others. Making a PR to fix

This should be fixed! @strepanier03 do you mind testing when you have a moment? I might be able to get to it later this week

Yup, I can test. Doing it now and will update shortly.

I'm able to upload the files successfully and download them, but I don't have a program on my mac to open them. When I try it opens in ATOM, which isn't a vector file program.
The behavior looks updated and correct if I had a program to open an EPS file though.

Issue not reproducible during KI retests. (Second week)

Okay I think I'm gonna close this out!

@strepanier03 It is working perfectly on both staging and production. Can you send me an offer for the reporting bonus

My bad! Reopening to handle payment.

Thanks @dangrous - I'll work on the reporting payment and close out when it's wrapped.

@jo-ui - I can't figure out who you are in Upwork. Can you apply for the job here or link your Upwork profile and I'll hire you?

@strepanier03 I've applied for the job on upwork

Thanks @jo-ui - I have hired you again and will keep an eye out for your acceptance so I can pay this and get this close out today.

Thank you @strepanier03 I've accepted the offer on upwork

I've payed out the contract, ended it, and left feedback.
Thank you @jo-ui for being part of this community 🙌


vue init or any command (on 2.9.4, not 2.9.3) , shows an error message

Version
2.9.3
Reproduction link
https://gist.github.com/akarelas-pt/1d0a4c86952302d585fc7ddeb5db7827
Steps to reproduce

npm uninstall -g vue-cli
npm i -g vue-cli
vue init webpack my-test

What is expected?
The init command should work without error messages
What is actually happening?
The init commands doesn't work, and I get the following 2 error messages:
$ vue init webpack my-test
Unknown command init.
vue-cli · Failed to download repo vuejs-templates/[object Object]: Response code 404 (Not Found)

The bug-submission form doesn't let my choose v2.9.4 (there's no such option), so I chose 2.9.3 in the selection box.

I also encountered the same problem, and finally chose the 2.9.3 version :)

I got the same issue, this is very unforgivable, not cool.

Duplicate of #1413


[BYOC][TensorRT] Fix dynamic batching when use_implicit_batch=False

Fixes error for subgraphs with dynamic batch size when use_implicit_batch=False. Added test case.
Also fixes deserialization of cached engine when batch size is not 1.

Thanks @trevor-m


does not work on macOS Monterey

when trying to run vagrant up
There was an error while executing VBoxManage, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
Command: ["hostonlyif", "create"]
Stderr: 0%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to create the host-only adapter
VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterfaceWrap, interface IHostNetworkInterface
VBoxManage: error: Context: "RTEXITCODE handleCreate(HandlerArg *)" at line 95 of file VBoxManageHostonly.cpp

You have to allow Virtual box extension in Privacy settings and reboot system.


What is the core code of unsupervised embedding of SGPT?

@Muennighoff Thank you very much!

Essentially this script: https://github.com/Muennighoff/sgpt/blob/main/crossencoder/beir/sgptce.py

But it's equivalent to this simplified code: https://github.com/Muennighoff/sgpt#asymmetric-semantic-search-ce
The py script I sent above just has some additional batching and benchmarking overhead


broken disk check is wrong in some case

Describe the bug
cfs_master_disk_error show a disk is wrong, but it's ok in fact.
if two machine's disks with the same disk path broken at the same time,  one disk_err warn will always exist, even if two disk both recover health.

fixed by #1973
commit: a9d300baa0bd884e27bb4eeb1b0368e016abfb20


Test only tests for old getUserMedia API

This won’t change much as browsers that are unprefixed support the new
promises based API.
It will show the difference that other implementations have not updated
if the PR is accepted which reduces points for only supporting prefixes.
Fixes #432

Thanks for the PR! I fixed this in a slightly different way, so I'm closing this.


E2E

End to end tests for Juno:
This PR includes tests for every function of Juno.

This failed because of we don't have v11-init-chain image. Automate workflows to create it will run when we merge this PR into main.

ok so this is blocked by:

https://github.com/CosmosContracts/juno/issues/371

?

@faddat yep

Let's slate this for v13 then, no worries :)

I prefer using old log. Easier for debug.

Is this still blocked?

@JakeHartnell it never really was, Jack had wanted to use ibctest.  I defer to @vuong177 and @Reecepbcups as to weather that is the right path.  I need to watch the ibctest presentation.

I think we should merge this PR first. We can integrate with IBC-test by future PRs. By #371 has been merged, this PR is ready to merge.

Thanks vuong!  I am going to defer to others, on this, but will enable auto-merge.

This branch should add some attribution to gaia/osmosis.

Will add a README.md about this.

I think this PR can be merged ASAP. It's working pretty well now with some test ( like Chain-upgrade, IBC transfer ). I know we are planningibc-testintegration but we can use e2e first in this time

I've removed the TODO and am adding an issue, goal should be to keep statik out of our code, and use embed to serve the OpenAPI UI, which is very nice.

https://github.com/CosmosContracts/juno/issues/400


Passing autoformat as boot parameter causes kernel panic

RancherOS Version: 0.6.1
Where are you running RancherOS? (bare-metal and virtualbox)
When passing the following parameters rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/sda] at boot time with an empty /dev/sda I get a kernel panic.
I tried multiple bare-metal machines and a VirtualBox VM using PXE as described in the manual.
Here is the panic message from the VirtualBox VM.

The issue only occurs if an autoformat takes place. It doesn't happen if the state partition is already there and it doesn't happen either if the rancher.state.dev=LABEL=RANCHER_STATE parameter is omitted.

It looks like the PXE docs were missing the rancher.state.formatzero flag. I updated the page to include this. If you add this flag to your setup then this should resolve the issue for you.
At the same time, it's definitely a bug that a kernel panic happens in this scenario.

@joshwget Thanks I will give it a try

@joshwget After adding rancher.state.formatzero I'm still getting the kernel panic.

Is the device in rancher.state.autoformat wrong perhaps? This was something I accidentally did when I was testing this yesterday.

I used
rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/sda] rancher.state.formatzero rancher.cloud_init.datasources=['url:http://ipxe.lan/cloud-config']

If I boot with only
rancher.cloud_init.datasources=['url:http://ipxe.lan/cloud-config']

and then access /dev/sda I can see it and it contains no partitions or filesystems (I used wipefs before)

The kernel panic here should be fixed by #1252. In v0.7.0 it's likely that state will still fail to be mounted, but at least we'll be able to boot RancherOS running from memory. This will make it easier to troubleshoot.
If you'd like, you can try this out with v0.7.0-rc2.

I'm having this exact issue.

@ChromoX It's on our releases page.

Is there anyway to hook this up to iPXE easily? iPXE doesn't seem to support https urls.

Github supports HTTP for downloads, so you can just switch it to http:// for testing.

Github sends a HTTP 302, which seems to force a redirect to https which iPXE fails on. I guess I'll have to stand up a HTTP server.

So I have RancherOS 0.7.0-rc2 loading up, but autoformat doesn't seem to work.
My cloud-config contains:
rancher:
  state:
    fstype: auth
    formatzero: true
    dev: LABEL=RANCHER_STATE
    autoformat:
      - /dev/sda
      - /dev/sdb

[root@compute3 ~]# sudo system-docker ps -a
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                        PORTS               NAMES
83058ba4a310        rancher/os-preload:v0.7.0-rc2     "/usr/bin/ros entrypo"   27 minutes ago      Exited (0) 27 minutes ago                         preload-user-images
59f7da69a58a        rancher/os-docker:1.12.1          "/usr/bin/user-docker"   27 minutes ago      Up 27 minutes                                     docker
35ca66d9b57d        rancher/os-console:v0.7.0-rc2     "/usr/bin/ros entrypo"   27 minutes ago      Up 27 minutes                                     console
be460fbede0b        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Up 27 minutes                                     network
495dd5a86c10        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Exited (0) 27 minutes ago                         cloud-init-execute
43fb3dd490a8        rancher/os-cloudinit:v0.7.0-rc2   "/usr/bin/ros entrypo"   27 minutes ago      Exited (0) 27 minutes ago                         cloud-init
dfaf74ef3dec        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Up 27 minutes                                     ntp
80d5347f6f30        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Exited (143) 27 minutes ago                       network-pre
592b57e97180        rancher/os-cloudinit:v0.7.0-rc2   "/usr/bin/ros entrypo"   27 minutes ago      Exited (0) 27 minutes ago                         cloud-init-pre
5c0f91e5ee83        rancher/os-udev:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Up 27 minutes                                     udev
dac1c4bdf3c9        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Created                                           all-volumes
861bfe67f341        rancher/os-udev:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Exited (0) 27 minutes ago                         udev-cold
38c9e425f4f8        rancher/os-preload:v0.7.0-rc2     "/usr/bin/ros entrypo"   27 minutes ago      Exited (0) 27 minutes ago                         preload-system-images
980aafc6557b        rancher/os-acpid:v0.7.0-rc2       "/usr/bin/ros entrypo"   27 minutes ago      Up 27 minutes                                     acpid
d944fb996a68        rancher/os-syslog:v0.7.0-rc2      "/usr/bin/ros entrypo"   27 minutes ago      Up 27 minutes                                     syslog
134335746e22        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Created                                           container-data-volumes
c1e620e19aa1        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Created                                           user-volumes
8207cdd8d2cc        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Created                                           system-volumes
d6ddbde3be18        rancher/os-base:v0.7.0-rc2        "/usr/bin/ros entrypo"   27 minutes ago      Created                                           command-volumes

Images:
[root@compute3 ~]# sudo system-docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
rancher/os-console       v0.7.0-rc2          10721c426834        4 days ago          25.52 MB
rancher/os-udev          v0.7.0-rc2          68d6ea8fe83f        5 days ago          25.51 MB
rancher/os-syslog        v0.7.0-rc2          953158db3d0b        5 days ago          25.51 MB
rancher/os-statescript   v0.7.0-rc2          7d58f531dea7        5 days ago          25.51 MB
rancher/os-preload       v0.7.0-rc2          c22e22148a3c        5 days ago          25.51 MB
rancher/os-cloudinit     v0.7.0-rc2          3c9eae2aa014        5 days ago          25.51 MB
rancher/os-autoformat    v0.7.0-rc2          498c83fc5816        5 days ago          25.51 MB
rancher/os-acpid         v0.7.0-rc2          b8f764b533d1        5 days ago          25.51 MB
rancher/os-base          v0.7.0-rc2          bc0fafb50f8f        5 days ago          25.51 MB
rancher/os-docker        1.12.1              ccc1cfe70218        6 weeks ago         98.19 MB

If I try to run the os-autoformat image:
[root@compute3 ~]# sudo system-docker run rancher/os-autoformat:v0.7.0-rc2
+ MAGIC='boot2docker, please format-me'
+ DEVS=(${AUTOFORMAT})

If I try to run the os-cloudinit image:
[root@compute3 ~]# sudo system-docker run rancher/os-cloudinit:v0.7.0-rc2
system-docker: Error response from daemon: Container command '/usr/bin/ros' not found or does not exist..

Any ideas?

@ChromoX How are you setting your cloud-config? rancher.autoformat is designed to be used as a kernel parameter.

@bachp I confirmed that a kernel panic no longer occurs in v0.7.0-rc4. I'm going to leave this issue open since we still haven't resolved why RancherOS was unable to mount state in your case.

@joshwget You were right. Wasn't using it as a kernel parameter. Thanks!

We'll probably support autoformat as part of a cloud-config at some point in the future after #1175.

@bachp Can you try this on v0.7.0 when you have the chance?

@joshwget I retried with v0.7.0. But I'm unable to perform autoformat. I tried the instructions at: http://docs.rancher.com/os/storage/state-partition/ and I also tried adding rancher.state.formatzero.
I also tried to add add autoformat to cloud-init as described here: http://docs.rancher.com/os/storage/state-partition/ again without luck.
I used wipefs -a /dev/sda on the disk before. Also I see an "autoformat" job starting but nothing seems to happen.

Same here. With 0.7.0 autoformat does not work when booting via IPXE.
Kernel params: rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/vda]
Couldn't find anything interesting in the debug logs other than this:

These seem like two separate issues. For @bachp the system is booting but autoformat isn't running.
@janeczku Do you have any other kernel parameters?

@joshwget  I'm also booting via iPXE. So the issue might be related to the one reported by @janeczku .
I used the parameters as described here: http://docs.rancher.com/os/running-rancheros/server/pxe/ no additional kernel args.

@joshwget

$ cat /proc/cmdline

rancher.state.dev=LABEL=RANCHER_STATE rancher.state.autoformat=[/dev/vda] rancher.cloud_init.datasources=[ec2]

in my case, the new, un-formatted, new disk wasn't all zeros, so the autoformat didn't trigger. for my testing use, I think I'd like a rancher.state.autoformat.onboot=true

I got this when I set sudo ros config syslinux, set the parameters, then reboot. I'd been messing with the parameters before that though.


ISSUE: Toggling Enable does not trigger code slot synchronization

Describe the bug
(using slot 1 as an example). Toggling input_boolean.enabled_frontdoor_1 does not trigger the synchronize_codeslot_frontdoor_1 automation.  Toggling the current day of the week does trigger the synchronization.
Environment (please complete the following information):

OS: HassOS
KVM on ubunto 20.04
Home Assistant version: 2021.9.3
Hassio
Component version: 0.0.63
Z-Wave Integration: zwave_js with zwave2mqtt community addon
Lock make and model: Schlage BE469

Logs
There are no errors.

I am unable to replicate this issue now.  Perhaps this is due to a homeassistant restart, not sure. Closing.


Various(not English) subtitles

Hello! I've tried to download Russian-language courses(e.g. https://www.coursera.org/learn/ekonometrika/home/welcome), but when I downloaded them I realised that there are no subtitles. Can you get me an advice? What do I need to do to download a course with Russian subtitles?

I will add support for downloading Non-English subtitles in the next release.


Warning: Ignoring unknown fps=""

Describe your bug here.
Having an issue when i'm compiling, whenever i try using lime test windows, it just writes "Warning: Ignoring unknown fps="" then does nothing for a good 2 minutes and gives me the prompt back.
This is the newest version, just downloaded it, already ran setup-windows.bat, installed all the needed libraries,
Any idea on how to fix this?
Command Prompt/Terminal logs (if existing)
C:\Users\blank\Documents\FNF psych engine>lime test windows
Warning: Ignoring unknown fps=""

C:\Users\blank\Documents\FNF psych engine>

Are you modding a build from source or with Lua?
Source
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No, fresh install.

"Ignoring unknown fps" is normal, but I really don't understand why lime isn't doing much when compiling

how to fix it?


Development enhancements

Development enhancements:

[ ] Lint support
[ ] .editorconfig for IDE
[ ] Pre-commit hook for linting

@shahnawaz I would be happy to help set this up. Did you have any ideas as far as linting packages or a code style guide you would like to implement? I personally like to use eslint with Airbnb's linter package eslint-config-airbnb-base. Also, I have previously used Husky - Git Hooks for pre-commit hooks. If you are fine with this, I will create a pull request!


The nullable attribute in the shema not be handled when creating the table with pyspark

pyspark version:3.2.0
spark-clickhouse-connector:0.4.0
code：

output:

It's by design https://github.com/apache/spark/pull/25536

It's by design apache/spark#25536

But if this specification is followed, a not null clickhouse table cannot be automatically created using the spark API, and if the table is created in advance, it cannot be replaced because the metadata in zookeeper is different

I am also confusing w/ this behavior, but unfortunately, Spark does not support reserving the nullability of data when using CTAS.

Opened https://github.com/apache/spark/pull/41070 to try solving this problem on upstream


Warehouse does not reconcile

Checklist

[x] I've searched the issue queue to verify this is not a duplicate bug report.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of kargo version.
[x] I've pasted logs, if applicable.

Description
This is a continuation of this issue. I continue to experience warehouses not respecting spec.interval or reconciling when changed. I've observed the same behavior across ~20 different projects. It does not seem related to git/image repo because when the reconcile does finally trigger, it executes in a matter of seconds, as can be seen in the screenshot of logs. The reconcile occurs at fairly regular interval of close to 15-20 minutes when the interval is set to any time, ranging from 2m to 10m. Triggering a sync via gui adds annotation to warehouse, but even this does not trigger the reconcile. It eventually reconciles on the same interval.
Screenshots

Steps to Reproduce

Create a warehouse with spec similar to

spec:
  freightCreationPolicy: Automatic
  interval: 2m0s
  subscriptions:
    - git:
        branch: main
        commitSelectionStrategy: NewestFromBranch
        discoveryLimit: 20
        includePaths:
          - regex:path/to/manifests/.*
        repoURL: https://github.com/<org>/<repo>
    - image:
        allowTags: ^production-[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z-[0-9a-f]{40}$
        discoveryLimit: 20
        gitRepoURL: https://github.com/<org>/<repo>
        imageSelectionStrategy: Lexical
        repoURL: xxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/<repo>

Click refresh on the warehouse in gui
Watch it spin until it eventually reconciles

I'd expect immediately after clicking reconcile, the modified warehouse would be seen by the controller and reconcile - triggering a freight fetch.
Version
Client Version: v1.0.3
Server Version: v1.0.3

@wmiller112 first, my apologies. I really thought we'd found it and nailed it in #2544 and #2754.
I know you don't suspect rate limiting at this point (nor do I), but I'm still curious if you observe any change in this behavior when you're subscribed to just one repo. In particular, if you subscribe only to the Git repo, does the situation improve?
Also, just to get more data here, is there any sharding involved here?

It may be an idea for us to bake an image with net/http/pprof enabled so we can collect a profile.

Its the same for a warehouse with only a git subscription.
I've built/am running the controller image with pprof, and I'm able to collect profiles/traces, but am not exactly sure what to look for. Also added some additional logging around requeue interval. Confirmed requeue is being set to defined interval when reconcile does run.

The things typically of interest are the following endpoints:

/debug/pprof/goroutine?debug=2 to see what goroutines are currently running, waiting, or blocked.
/debug/pprof/profile to see what the CPU is spending most time on.
/debug/pprof/heap to get a better insight into memory allocations.
/debug/pprof/trace?seconds=<number of seconds> to collect a trace (i.e., almost anything the program is doing over these seconds).

Some of these profiles can hypothetically contain slightly sensitive data (e.g., memory addresses or minimal operational details). If you want to share them, you can do so via mail<EMAIL_ADDRESS>

My current working theory based on the traces shared by @wmiller112 in private, is that due to the concurrency configuration of our reconcilers currently being set to the default (which equals 1). Things are not being picked up in time because the controller is working on gathering the latest image build information for another Warehouse (which can be quite an expensive task depending on the number of images you have).
I have shared a branch with him (https://github.com/hiddeco/kargo/tree/allow-concurrency-config) that allows the configuration of the number of concurrent reconciliations to see if this proves I am right. If this is true, we should create a proper solution for this and think about the default configuration we want to ship with.

With the MAX_CONCURRENT_RECONCILES set to 4 I definitely see more timely warehouse reconciles (down to about 5-10 minutes, with the interval still set to 2m). Interestingly the execution of the reconcile takes longer (was 10-20 seconds, after at about 1-2 min). These logs are for single warehouse, same as initial post:

It seems there is a point of (completely) diminishing returns; Increasing beyond 4 (tried 6, 8) reconcile seems to start on some warehouses and just hang for 20+ minutes. Have been keeping an eye on resource utilization and staying well within requests/limits

Would it be possible for you to send me another set of profiles with this configuration enabled?

After some more debugging through private channels, the reason for this can be summarized as follows:

At present, all reconcilers have a single worker configured. Because of this, anything that takes a long time will delay the processing of all other resources of the same kind.
When using (many) "newest build" selectors for container images in Warehouses, there is a high possibility they will run into rate limits from the registry at some point. Because there is a fair chance multiple Warehouses will be rate-limited due to the request coming from the same IP, the credentials being shared with other Warehouses, the rate limit being tied to the account/organization, or a combination — it will cause a flood of HTTP 429 responses. Combined with our fairly simple registry implementation, I suspect these are not detected correctly and retried, in addition, I suspect that running Goroutines that have already kicked off are not canceled as soon as a first error occurs.
We are not configuring GOMAXPROCS and GOMEMLIMIT to match CPU and memory limits while this is required for Go to properly detect the limits of the Pod, and not the limits of the host:

https://blog.howardjohn.info/posts/gomaxprocs/
https://www.ardanlabs.com/blog/2024/02/kubernetes-cpu-limits-go.html

To address this, we should take multiple actions:

[ ] Allow configuration of the number of workers, preferably per reconciler. The rationale for this is that not every kind is expected to be a long-running process. Warehouses and Promotions can be, but Stages should be relatively quick.
[ ] Ensure the chart properly detects CPU and memory limits and configures the earlier-mentioned environment variables. For example:env:
- name: GOMEMLIMIT
  valueFrom:
    resourceFieldRef:
      containerName: controller
      resource: limits.memory
- name: GOMAXPROCS
  valueFrom:
    resourceFieldRef:
      containerName: controller
      resource: limits.cpu

[ ] Revise the registry logic, find the gaps that result in a performance impact, and seal them so it won't happen again.

Hi, I am on 1.0.3 and I observing the similar behavior.
I have 1 project, 1 warehouse, 1 stage and it does not discover new docker images automatically... If I press button to refresh, it will find it but other otherwise it does not do anything.
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: testing-playground-warehouse
  namespace: app
spec:
  interval: "1m0s"
  subscriptions:
    - image:
        repoURL: registry.gitlab.com/xxxxxxx/testing-playground-service
        strictSemvers: false

I deleted Project, Warehouse and Stage and recreated them again an now it works fine. Will try to do some stuff to get it stuck again.

Just a note, I am observing this behavior again. I still have just 1 Project, 1 Stage, 1 Warehouse.
I dont see any errors except:
time="2024-10-29T14:40:32Z" level=error msg="finished streaming call" connect.code=internal connect.duration=17.922126759s connect.method=WatchWarehouses connect.service=akuity.io.kargo.service.v1alpha1.KargoService connect.start_time="2024-10-29T14:40:14Z" error="internal: context canceled"
time="2024-10-29T14:40:32Z" level=error msg="finished streaming call" connect.code=internal connect.duration=17.92214805s connect.method=WatchStages connect.service=akuity.io.kargo.service.v1alpha1.KargoService connect.start_time="2024-10-29T14:40:14Z" error="internal: context canceled"

So if it would be Rate limit I would expect to see some errors in pods logs. I dont really have reproducible steps yet, the only thing I was doing is changing Stage manifest (testing different imageSelectionStrategy
Let me know if you think this is completely different issue, I can open new one.

@ddeath you're looking at API server logs when you should be looking at controller logs.

@krancour I checked all pods (kargo-api, kargo-controller, kargo-management-controoler, kargo-webhooks-server) and the only errors there where those from kargo-api

@ddeath, you'll notice @hiddeco and @wmiller112 had a lot of back and forth over the course of two issues to figure out what was going on here. @hiddeco has summarized findings nicely above. You and I also had some recent discussion about what sort of things can make Warehouse reconciliation slow (and therefore become a bottleneck). Please try to use this information to determine whether you may have one or more poorly-configured Warehouses that are taking a long time to reconcile and blocking others from reconciling at regular intervals. Enabling debug or even tracke-level logging may help as well.

Yeah I am slowly doing that. However as I noted I have only 1 Warehouse with only 1 subscription (with private token, not public one so rate limits will be higher) and I am not seeing any reconcile happening for a long time (30 minutes). So  just reporting my findings 🤷
But ok I promise will not write another comment with some sort of hard evidence of what is wrong in my case...

with only 1 subscription

What are its details? Is it a subscription you'd expect to perform poorly based on our other discussion?

Is anyone still experiencing difficulties with this in v1.1.0 or greater?


docs(protocol): Correct some offsets in Server Interaction

Offsetting codes are fun. Correct the codes in the protocol.rst by referring to the table.

@ProbablePrime,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Promscale has been discontinued

Dear community,
We regret to inform you that we’ve made the difficult decision to discontinue Promscale.
Promscale was created to help the community use PostgreSQL/TimescaleDB as a fully compliant remote storage for Prometheus. Later on we added OpenTelemetry and Jaeger tracing support. With its full SQL support, it provided a unified storage for traces and metrics with deep analysis and correlation capabilities. We also built tobs to simplify the deployment of a complete observability stack with one command.
Promscale has been a labor of love for our team at Timescale  for over two years, and we’re grateful for the support and feedback from you, our community of users. However, after reevaluating our company priorities, we’ve decided to redirect our resources to other product areas.
We understand this news may be disappointing, so please see the FAQ for answers to common questions including alternative solutions.
Thanks again for your support.
FAQs
Why is Promscale discontinued?
We’re re-focusing all our efforts on the core of what we do best, creating a great developer experience for time-series application developers.
What happens next?
The following Github repositories will remain publicly available but will no longer be evolved nor maintained:
https://github.com/timescale/promscale
https://github.com/timescale/promscale_extension
https://github.com/timescale/tobs
https://github.com/timescale/helm-charts/tree/main/charts/promscale
We’ll continue to provide support for critical issues to existing customers of Promscale with Timescale Cloud or MST until April 30th, 2023, at which point support will end.
Can I keep using Promscale with TimescaleDB?
You can continue to use Promscale with TimescaleDB although we don’t recommend it since Promscale will no longer be supported nor maintained.
The Github repositories and release artifacts will remain publicly available. We can’t ensure that Promscale will continue to work with newer versions of TimescaleDB and PostgreSQL.
If you want to make modifications, we encourage you to fork the repositories.
Can I keep using the open-source version of Promscale with Timescale Cloud / MST?
You can until April 30th. Before April 30th you should transition to a different solution since we can’t ensure that Promscale will continue working and we plan to remove the unsupported promscale extension from our servers.
Who do I contact if I am a customer of Timescale Cloud / MST and I need critical support?
Contact the support team at<EMAIL_ADDRESS>or https://www.timescale.com/support
What Promscale alternatives do you recommend?
There are different options depending on your use case.
If you use Promscale as a remote storage for Prometheus there are a number of other options you can consider. There is a list provided in the Prometheus documentation.
If you are using Promscale as a storage system for your OpenTelemetry or Jaeger traces there are a number of options you can consider. This is a list of products that have native support for OpenTelemetry. You can use the OpenTelemetry Collector to convert your Jaeger traces to OpenTelemetry traces.
If you have a specific need, you can ask in the #promscale channel in the Timescale slack community and we’ll try to help you.

sad.. it was fun using it. sad to see it goes.. but for the best for the team.. all the best..

But this is the best storage solution for self-hosted Prometheus metrics 😭

Hey, I'm sorry to see this discontinued, but I understand the need for hard prioritization decisions. It was always a pleasure seeing Promscale at Prometheus Day and KubeCon events.
Just to clarify, is TimeScale shifting away from observability data altogether? Or just away from Prometheus compatibility?

Thanks, @aantn.  This announcement only affects Promscale.
Timescale will continue to evolve as a flexible and powerful database that can be used for a number of different use cases.

Hi, we are currently researching to create an observability stack and would like to use TimescaleDB as its storage. However, it seems that now as Promscale is deprecated, there is no part left that would connect either Jaeger / OTEL / Prometheus with Timescale. How could we achieve this?
As a side remark: when searching for connecting OTEL with Timescale, the results are still full of mentions of Promscale. In addition, there are articles in the Timescale blog that refer to Promscale without even mentioning its deprecation. The deprecation notice on the top of the promscale does not mention a sane replacement (neither are the alternatives mentioned above really useful). IMHO deprecating Promscale looks like cutting off Timescale from a multitude of use cases involving OTEL / Jaeger / Prometheus. In our case, we might have to look elsewhere for a storage if we cannot find a replacement for Promscale, even though we have compelling reasons to prefer Timescale to all the other considered alternatives.

Bummer, we have a significant amount of custom reporting that pulls data out of the hypertables. Not trivial to replace with Mimir. Any chance of a community maintained fork happening?

Hey, can anyone suggest a method to connect node exporter to TimescaleDB?


Strip off links to obsidian documents

Discussed in https://github.com/mvdkwast/obsidian-copy-as-html/discussions/17

Originally posted by aravindk100 January 12, 2023
Can you add an option to strip off links and convert them to plain text - Links are for other notes, which does not mean much when I copy and paste into another software like Outlook?

Similiar issue are also on footnotes.

Thanks for reporting @karlla1220, I moved this to a new issue for this. Feel free to complete there how you think that footnotes should be handled.

Thanks for implementing this, did not even realize this was implemented until now.


Add embedded graphs to SyncManager

From<EMAIL_ADDRESS>on November 06, 2013 09:22:42
Cr2bug currently shows graphs for sync # of updates and bandwidth use.  These should be made generic and embedded into the SyncManager model.
Original issue: http://code.google.com/p/foam-framework/issues/detail?id=44

Is this still relevant?

obsolete