Add support for user defined functions

Possibly denoted by uppercase characters

There is a small implementation problem. How will I store user defined functions? The context HashMap is already kind of cumbersome to have to pass in to all the functions, so I don't want to create another func_context or something.

Possible solutions:
Store exactly the same as other variables
// Pseudo code
context: HashMap = [
    ("F", "x^2 + x + 1"),
    ("G", "\sin(x)"),
];

This would probably rule out multi-variable functions, would force all functions to be uppercase (or another distinguishing feature), and would require the free variable to be x (or another specific letter/identifier)
Store with an enum as a key
// Pseudo code
enum VarEntry = {
    Var(String), // Name
    UsrFn(String, Vec<String>), // Name, variables
}

use VarEntry::*;
context: HashMap = [
    (UsrFn("F", ["x", "y"]), "\sqrt(x^2 + y^2)"),
];

I don't think there are any drawbacks. But, I will need to make a way to index them (Using name probably)
(I just realized this might be a bad idea)
Store with String as key, enum as value
context: HashMap = [
    ("A", UsrFn(["x", "y"], "\sqrt(x^2 + y^2)"))
]


Remove dsn.lower() from influxdb08.InfluxDBClient.from_DSN()

InfluxDBClient.from_DSN() now behaves correctly and allows (for example) case-sensitive password.

Thanks for spotting that. Could you add to the influxdb08 package the similar tests that we added to the main package? https://github.com/influxdb/influxdb-python/blob/master/tests/influxdb/client_test.py#L810
Thanks.

In my opinion these kinds of problems are easiest to capture by using the existing tests, but changing the case-sensitive fields (username and password, others?) to include values which use uppercase letters.
So for example tests/influxdb/influxdb08/client_test.py#L102 would change from
cli = InfluxDBClient.from_DSN('influxdb://usr:pwd@host:1886/db')
to
cli = InfluxDBClient.from_DSN('influxdb://uSr:pWd@host:1886/db')
Also, for easier change (and avoiding typos) the "base url" should be one variable, instead of rewriting it all the time (e.g. DSN_STRING = 'influxdb://uSr:pWd@host:1886/db'). This way the same "base url" could be used in all the cases, just by using 'udp+' + DSN_STRING for UDP-test etc.
I can try and do this today or tomorrow, but if this is the way to go, I'd rather create another pull request, as it affects multiple tests (I'd also do the changes to 0.9 at the same time).

Just added some refactoring of the tests, which also modify the tests to include the situation this pull request fixes.

Good point, changed as requested :)

Thank you for your contributon @kipe !
@ChristopherRabotin  Is that good to you?

Yes, that looks good. Thanks for asking.
On Mon, Jun 1, 2015, 19:22 Alexandre Viau<EMAIL_ADDRESS>wrote:

Thank you for your contributon @kipe https://github.com/kipe !
@ChristopherRabotin https://github.com/ChristopherRabotin Is that good
to you?
—
Reply to this email directly or view it on GitHub
https://github.com/influxdb/influxdb-python/pull/190#issuecomment-107662767
.


Keybinds for extensions

It would be nice to have specific keybinds for extensions or disabled them for some. I have some thoughts already, but this may be a limitation of WezTerm as it is

A thought is to have the option to specify a key table that gets activated when the extension is active. That way I can avoid having to pass the config object to the plugin directly. The hard part with this is that I would have to make it work with the mode component to either create a new mode that is modified by the extension or just ignore it and overwrite the normal_mode that is currently being done. Maybe similar to how the plugin expects the key_tables that represnet a mode to start with mode_ an extension keytable can be expected to start with ext_. Still having some thoughts about this

Honestly my use for this would be to have some empty key tables when I use the fuzzy finders for resurrect, workspace switcher or domains so the normal keybinds like switching tabs are disabled. Since I dont want to be able to switch away from the fuzzy picker when it is open since that can be confusing with the tabline being altered


URL field is locked

I've implement the control on the Account entity, but it locked the field. I can't type any text in the field.

What happens when you don't specify the API Key?

Its working fine now.
Thanks,


LangChainDeprecationWarning when using ChatAnthropic and llm-term gets stuck.

When using llm-term with the Anthropic provider and the claude-3-opus-20240229 model, I encountered the following issues:

LangChainDeprecationWarning: After running llm-term -p anthropic -m claude-3-opus-20240229, I received the following warning: /home/user/.local/pipx/venvs/llm-term/lib64/python3.11/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class langchain_community.chat_models.anthropic.ChatAnthropic was deprecated in langchain-community 0.0.28 and will be removed in 0.2. An updated version of the class exists in the langchain-anthropic package and should be used instead. To use it run pip install -U langchain-anthropic and import as from langchain_anthropic import ChatAnthropic.

llm-term gets stuck: After entering a prompt, llm-term gets stuck and does not provide any response (keeps loading the bar progress).

Steps to Reproduce

Install llm-term with the anthropic extra: pipx install "llm-term[anthropic]"
Add environment variable ANTHROPIC_API_KEY
Run llm-term -p anthropic -m claude-3-opus-20240229
Enter a prompt

Environment

Operating System: [Linux opensuse tumbleweed]
Python Version: [3.11.8]
llm-term Version: [0.12.0]

Additional Information
With OpenAI I don't have any issues and I can do llm-term -m gpt-3.5-turbo-0125 and llm-term -m gpt-4-0125-preview and it works like a charm.  (Oh by the way, I tried as well with another model with anthropic and it gets stuck as well)

Oh shoot, sorry about that. I will get this squared away

Hey @theDataFixer PyPI is having a security incident right now and I'm getting an error in CI/CD uploading the latest release:
Uploading llm_term-0.13.0-py3-none-any.whl
25l
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/19.8 kB • --:-- • ?
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/19.8 kB • --:-- • ?
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 kB • 00:00 • 20.8 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 kB • 00:00 • 20.8 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 kB • 00:00 • 20.8 MB/s
[25](https://github.com/juftin/llm-term/actions/runs/8462980902/job/23185155392#step:7:26)hWARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
         New uploads are temporarily disabled. See
         https://pypi.org/help/#admin-intervention for more information.

I will try re-uploading tomorrow morning but I believe the fix is ready.
It's worth noting that I'm not experiencing the same "llm-term gets stuck" issue you described above - I can debug that further if this upcoming fix doesn't resolve that.

Alright, llm-term 0.13.0 is released to PyPI. Give it an upgrade and let me know if the issues aren't resolved @theDataFixer

@juftin it works for models haiku and sonnet! However, for opus I got an error:

My prompt:

The models (https://docs.anthropic.com/claude/docs/models-overview):

Great news (that it works now). For the error, it looks like there was an overloaded APIStatusError on Anthropic's side with opus - looks to be resolved now

I am going to mark this issue as resolved for now - let me know if you face any more issues 😄


SSL database connection

Hi Everybody,
This is not an issue but a simple question.
Does Tedious support  SSL for Secure Connections ? I enabled SSL on my microsft sql server, but I got the error  read ECONNRESET
Thank you !!

works with options.encrypt =  true


Checkstyle errors

mvn clean install  fails with a checkstyle error about unsused imports:
[INFO] There are 2 checkstyle errors.
[ERROR] TravelTimePrediction.java[30:8] Unused import - org.apache.flink.api.common.typeinfo.TypeHint.
[ERROR] EventTimeJoinFunction.java[25:8] Unused import - java.util.Iterator.

Thanks for the pull request. We've added travis to prevent this from going unnoticed in the future.


Improve support for cast storage with local bytecode match

Component
Cast
Describe the feature you would like
From the docs description of cast storage:

Emit the slot number to get the full storage layout (requires contract to be verified on Etherscan with a Solidity version > 0.6.5 or you must be in a Forge project with a local contract matching the deployed bytecode).

In practice, the local option for cast storage doesn't function very well. There are 2 main cases:

If the target contract is a proxy, the storage layout you want is the implementation contract's. But if the proxy is defined in your local project (even via import from openzeppelin), the fetched bytecode will get a local match and give you results based on the proxy's layout. This is in contrast to the etherscan lookup logic, which handles this pretty well since etherscan's API lets you fetch the implementation contract.
If the target contract has immutable variables or other changes to bytecode that occur during the constructor, the deployed bytecode probably won't find a match in your local artifacts, since the comparison checks for an exact bytecode match.

Suggested improvements:

Ability to supply a flag to cast storage that forces the etherscan lookup, in case the local lookup returns weird results like the ones described above. This one's easy to implement, and I'd be happy open a PR for this if there's interest.
Somehow have a fuzzier bytecode match in the local lookup - one that matches regardless of constructor-set values. I'm not sure what this looks like in practice.

Additional context
In case you're interested in reproducing the things that prompted me to open this issue, what I did was clone and build this repo: https://github.com/Layr-Labs/eigenlayer-contracts
And I'm trying to do mainnet storage lookups for contracts defined there. For example: cast storage 0x54945180dB7943c0ed0FEE7EdaB2Bd24620256bc, which points to a contract behind a proxy (StrategyBaseTVLLimits.sol)

Ability to supply a flag to cast storage that forces the etherscan lookup, in case the local lookup returns weird results like the ones described above. This one's easy to implement, and I'd be happy open a PR for this if there's interest.

@wadealexc  would you still be willing to contribute a PR for this improvement? thank you!

@wadealexc on a second thought - I used
cast storage 0x54945180dB7943c0ed0FEE7EdaB2Bd24620256bc --rpc-url $RPC_URL --etherscan-api-key $KEY

(which I see is used in https://github.com/Layr-Labs/eigenlayer-contracts too) and it is recognizing that is a proxy / fetch impl source. Probably I am missing something, but is this still an issue? Thank you
No matching artifacts found, fetching source code from Etherscan...
Contract at 0x54945180dB7943c0ed0FEE7EdaB2Bd24620256bc is a proxy, trying to fetch source at 0xdfdA04f980bE6A64E3607c95Ca26012Ab9aA46d3...
| Name             | Type                     | Slot | Offset | Bytes | Value                                                                          | Hex Value                                                          | Contract                                                                 |
|------------------|--------------------------|------|--------|-------|--------------------------------------------------------------------------------|--------------------------------------------------------------------|--------------------------------------------------------------------------|
| _initialized     | uint8                    | 0    | 0      | 1     | 1                                                                              | 0x0000000000000000000000000000000000000000000000000000000000000001 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| _initializing    | bool                     | 0    | 1      | 1     | 0                                                                              | 0x0000000000000000000000000000000000000000000000000000000000000000 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| pauserRegistry   | contract IPauserRegistry | 0    | 2      | 20    |<PHONE_NUMBER>6482466234919654390548124283612655712                                | 0x0000000000000000000000000c431c66f4de941d089625e5b423d00707977060 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| _paused          | uint256                  | 1    | 0      | 32    | 0                                                                              | 0x0000000000000000000000000000000000000000000000000000000000000000 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| __gap            | uint256[48]              | 2    | 0      | 1536  | 0                                                                              | 0x0000000000000000000000000000000000000000000000000000000000000000 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| underlyingToken  | contract IERC20          | 50   | 0      | 20    |<PHONE_NUMBER>411140055499929126197631081361136981764                              | 0x000000000000000000000000be9895146f7af43049ca1c1ae358b0541ea49704 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| totalShares      | uint256                  | 51   | 0      | 32    |<PHONE_NUMBER>9146970069200                                                        | 0x0000000000000000000000000000000000000000000002d547e32f07f01158d0 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| __gap            | uint256[48]              | 52   | 0      | 1536  | 0                                                                              | 0x0000000000000000000000000000000000000000000000000000000000000000 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| maxPerDeposit    | uint256                  | 100  | 0      | 32    |<PHONE_NUMBER>37316195423570985008687907853269984665640564039457584007913129639935 | 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| maxTotalDeposits | uint256                  | 101  | 0      | 32    |<PHONE_NUMBER>37316195423570985008687907853269984665640564039457584007913129639935 | 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |
| __gap            | uint256[48]              | 102  | 0      | 1536  | 0                                                                              | 0x0000000000000000000000000000000000000000000000000000000000000000 | src/contracts/strategies/StrategyBaseTVLLimits.sol:StrategyBaseTVLLimits |

@wadealexc ping in case you could answer above. thank you!


Poor error message when parameter "in" definition missing

It's an error if the "in" attribute is not defined for a request parameter. In version 2.0.28, this error produces an accurate message:
attribute paths.'/operations'(post).parameters.[param0].in is missing

But in version 2.0.29, an additional error message is produced. Which is not very good. It is confusing (is "in" missing or not?) and misleading (no, there are not 2 errors).
attribute paths.'/operations'(post).parameters.[param0].in is not of type `string`

Hi @kerrykimbrough thanks for reporting this, would you please share a minimal sample of definition when this happens?
Thanks

param-in-missing.json.txt

Hey @kerrykimbrough I did a quick local test with tags/v2.0.29 and current snapshot and I haven't been able to reproduce the message with the definition provided.  This is what I'm getting: [attribute paths.'/operations'(post).parameters.[param0].in is missing]

Sorry, I misstated the problem symptom. I have corrected the original description.

The problem is caused by a recent change to OpenAPIDeserializer at line 1592:
...
8ba9dd98e	( haoyu.guo	2021-03-23 19:39:39 +0800	1590)		String value = getString("in", obj, true, location, result);
d1a151ca7	( haoyu.guo	2021-03-23 18:41:19 +0800	1591)
2d129e714	(gracekarina	2021-10-12 16:20:11 -0500	1592)		if (!result.isAllowEmptyStrings() && value == null) {
8ba9dd98e	( haoyu.guo	2021-03-23 19:39:39 +0800	1593)			return null;
8ba9dd98e	( haoyu.guo	2021-03-23 19:39:39 +0800	1594)		}
...

Previously, the missing "in" error caused value == null and an immediate return. But now the check for isAllowEmptyStrings allows things to continue to report the additional bogus error.  I don't understand the purpose of isAllowEmptyStrings but it seems very inappropriate for such a global setting to allow a violation of the OpenAPI spec to be ignored.

It seems that ParseResult.isAllowEmptyStrings may have been added to resolve issue #1631. But using it here doesn't make sense. The "in" value is extracted using getString which will return an empty string without collapsing it to null. getString correctly returns null only when the value is missing or explicitly null.
In general, applying isAllowEmptyStrings directly to the result of getString appears to be a mistake, unless some additional special logic is needed to address a specific situation (like issue #1631). If so, this mistake seems to be repeated many times in OpenAPIDeserializer.

Hi @kerrykimbrough  There is a new option for parser to allow empty strings as valid,  you can always just set the option to false, to have the behavior from 2.0.28, but I think there is no need as I just merged a PR with a better validation in the parameters deserialization.
I hope this helps.

I don't think PR #1645 is an improvement. It doesn't even change the behavior originally reported, i.e. the extra misleading "not of type string" message. I guess what you're saying is that the extra message is not really a problem.

Sorry, my previous comment was mistaken. I have deleted it.


After can_be_root=false Can not add page with this page type anymore

Hi I have added $can_be_root=false to my Article page. After that when I click to add page there is a message that "Note: Some page types are not allowed for this selection". I choose under another page option but the page type is disabled and I can not create a page wtih this tempalete anymore.
Now the interesting thing is = My page name is Article - I have a class named Article extends Page Class. If I create and copy Article to ArticlePage Class and after dev/build - I can create Article and ArticlePage template from admin panel. If I delete the ArticlePage class and after dev/build then Article is disabled again..

Thanks for copying this over to CMS @cansozeri! https://github.com/silverstripe/silverstripe-cms/issues/2040


[FEATURE]  @SelfLift

It would be nice if there would be a possibility to generate override methods that replace the self return type of the superclass. That way boilerplate code for creating fluent methods with inheritance would be reduced without using ugly self bounding generics
Example:
class Foo {
    @SelfLift  //mark method as 'self liftable'
    public Foo method() {
        return this;
    }
}

@SelfLift  //ask lombok to generate lifted methods
class Bar extends Foo {}

would produce Bar as:
class Bar extends Foo {
    @Override @SelfLift
    public Bar method() {
        return (Bar) super.method();
    }
}

by "lifting" all methods of the superclass which are annoted with @SelfLift.
"self lifting" without marking superclass methods would be better to extend existing thirdparty classes that way. But i see no realistic chance to do this in a stable way without unwanted effects.
(Maybe i had bad design, but that feature would had helped me several times. I guess i miss some critical problems with that idea and it will never be realized. However, thank you for your great work at Lombok.)

Cool feature, but, no can do.
[A] this requires resolution, which is really difficult.
[B] self-types are in general a missing java feature, and this hack of generating casts is a lot more complex then it sounds. Now imagine generics are involved, for example.
[C] generally this is solved by way of generics, see for example how enums are defined, or Comparable. (Comparable<E extends Comparable<E>>). This has various disadvantages compared to a true self-type, but nevertheless it's what is common today.
But mostly the resolution thing.


Use --depth 1 when fetch

Issue: https://github.com/argoproj-labs/argocd-image-updater/issues/475
Using depth 1 when fetching will help speed up when fetching a very large history repo

Thank you for your PR, @yuyuvn, and apologies for the late reply.
Unfortunately, this PR modifies a component that we copy verbatim from Argo CD and then enrich with our own functions. We do not touch any of the existing functions, though. That means merging this PR would result in some potential regressions in the future.
I think the correct way would be to leverage the Argo CD repo server for any Git related operations. We are in the process of merging the Image Updater into Argo CD, and then we could make such a modification aligned with how Argo CD envisions it, too.

@yuyuvn thanks for the enhancement patch. Shallow fetch is now implemented in image-updater. For details see https://github.com/argoproj-labs/argocd-image-updater/blob/62b07d5771ce4bedd2c6f9dc98db5bd11f0b4848/ext/git/writer.go#L164
https://github.com/argoproj-labs/argocd-image-updater/blob/62b07d5771ce4bedd2c6f9dc98db5bd11f0b4848/ext/git/client.go#L68


[redhat-satellite] add 6.16 and updates

Dates updated to the latest release as the automatic detection does not seem to work correctly.

Thanks @heuwerk!


Document additional value for new enum

@andreiztm : Thanks for your contribution! The author(s) have been notified to review your proposed change.


StackOverflow Exception when using urllib.unquote

urllib.unquote('url') consistently causes StackOverflow Exception in IronPython 2.7.7
Sames code works fine in CPython 2.7 + 3.5
Can someone tell me if they can reproduce it?
IronPython 2.7.7
IronPython 2.7.7 (<IP_ADDRESS>) on .NET 4.0.30319.42000 (32-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib import unquote
>>> unquote('https://api.airtable.com/v0/APPKEY/TABLE%20READ/recwPQIfs4wKPyc9Dasda')
Process is terminated due to StackOverflowException.

Python 2.7.13
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib import unquote
>>> unquote('https://api.airtable.com/v0/APPKEY/TABLE%20READ/recwPQIfs4wKPyc9Dasda')
'https://api.airtable.com/v0/APPKEY/TABLE READ/recwPQIfs4wKPyc9Dasda'

Python 3.5
Python 3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 15:51:26) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import unquote
>>> unquote('https://api.airtable.com/v0/APPKEY/TABLE%20READ/recwPQIfs4wKPyc9Dasda')
'https://api.airtable.com/v0/APPKEY/TABLE READ/recwPQIfs4wKPyc9Dasda'

https://github.com/IronLanguages/ironpython2/blob/master/Src/StdLib/Lib/urllib.py#L1221

I believe this one was fixed since the 2.7.7 release (https://github.com/IronLanguages/ironpython2/commit/57f90c367b81c3752e7c848f6c95a9a0fbcd3f8f). If you use the urllib.py from the master branch it should work.


Fix issue  #4

Replace ConsoleChar obsolete method from latest beta

Hi @oschwab - thank you for the pull request.
I have decided to fix this another way for now, in order to maintain compatibility with both stable & beta branch. I will close this pull request for now (but eventually this is the fix that I will transition to after the ConsoleChar changes move to stable)

Hi @oschwab - thank you for the pull request.
I have decided to fix this another way for now, in order to maintain compatibility with both stable & beta branch. I will close this pull request for now (but eventually this is the fix that I will transition to after the ConsoleChar changes move to stable)


[EasyCodingStandard] Check-markdown fix is eating PHP and strict_types declaration

Consider this PHP file test.php:
<?php declare(strict_types=1);

function foo(): void
{
}

Now ecs behaves as expected:
$ vendor/bin/ecs check test.php  # no error detected
$ vendor/bin/ecs check test.php --fix # no change in the file is made

However using check-markdown on test.md with PHP code example (eg. when there is complete file in copy-paste form, thus including PHP opening tags):
# Example file

```php
<?php declare(strict_types=1);

function foo(): void
{
}
```

$ vendor/bin/ecs check-markdown test.md  # no error detected
$ vendor/bin/ecs check-markdown test.md --fix # file is changed, PHP declaration is eaten

Output is
# Example file

```php
<?php
function foo(): void
{
}
```

=> the declare(strict_types=1); part and one empty line was eaten.
When there is not declare_script declaration and just PHP open tags, they are eaten as well:
# Example file

```php
<?php

function foo(): void
{
}
```

Output:
# Example file

```php
function foo(): void
{
}
```

This is expected, as long markdown files make documentation less readable. The PHP informaiton is already in the markdown tag.
It would be nicer to have same output though :)

This is expected, as long markdown files make documentation less readable.

To clarify: My use case was to provide some example file in the readme, so that people can copy-paste the whole content. So having also the PHP opening tag is wanted, readability is not a concern in this scenario - the purpose of the PHP code is in this use case different.

The PHP informaiton is already in the markdown tag.

Not really - there isn't the strict types declaration.
Also - it was quite unexpected having the code changed even though there was not code rule failing when running without --fix. My assumption was it will behave similarly as the check command.

Copy-pasting code snippets from Markdown is not a good idea, as they're not validate nor tested. It's a string basically :)
The purpose of this command is to give code in Markdown style, nothing more.
Feel free to contribute PR with opt-in features that you propose though.


360 update reactrouter code snippet

Updated Breadrumb code snippet and code block to match iclink styling. Added IcLink to imports
Tell us, in a few words, what the changes are.
Checklist

[ ] I have manually accessibility tested any changes, if relevant.
[ ] I have raised this pull request against the develop branch

These updates will be reapplied once https://github.com/mi6/ic-design-system/pull/379 have gone through


Moving Caskroom from HOMEBREW_REPOSITORY to HOMEBREW_PREFIX - Permission denied

Since this issue bit a lot of users in the last days, instead of opening a new one, please comment on this one if the following fix does not work for you:
First of all, this problem should only happen if you followed the instructions from brew update to sudo chown root:wheel /usr/local.

Temporarily set the permissions on /usr/local:

sudo chown $USER /usr/local
2. Rerun the `brew cask` command that failed.
3. You may now **optionally** change the permissions on `/usr/local` back:

    ```bash
 sudo chown root:wheel /usr/local

--
This issue was first raised in https://github.com/caskroom/homebrew-cask/issues/24638.

We haven’t used the chief bug label in a while. This one deserves it.

@reitermarkus A suggestion: check if HOMEBREW_PREFIX is writable and if not print the message telling people what to chown.

was directed to this issue from #24819, but I am still experiencing the same issues identified in #24819 even after following instructions/suggestions indicated here.

I ran into this issue as well. What was the justification for moving the Caskroom? Doesn't it mean that merely installing homebrew-cask now requires sudo?

Doesn't it mean that merely installing homebrew-cask now requires sudo?

That has almost always been the case, except for a very short window when we had the Caskroom in HOMEBREW_REPOSITORY. Requiring sudo for install by default is nothing new.
That said, you can still change the location of the Caskroom by adding --caskroom=/path/to/Caskroom to your HOMEBREW_CASK_OPTS environment variable.

In truth, I actually liked that we stopped requiring sudo. But the more I think about it, the more I can see points in favour of both cases.
Having everything inside of Homebrew (the directory) makes /usr/local just a bit cleaner, does not require sudo, and it makes some sense to have Cellar and Caskroom together. On the other hand, both projects do behave differently and install different types of apps in different ways, so having Caskroom separate from Homebrew also makes some sense.
There’s also the matter of user experience. Installing Homebrew is something big we’re doing, so sudo seems acceptable. On the other hand, a user typing brew cask install google-chrome the first time and being prompted for the sudo password seems a bit weird (the former was explicitly a setup process, unlike the latter, though this all goes out the window when installing pkgs).
Considering the breakage having Caskroom inside Homebrew (due to the recent move) caused, I think it’s sensible to have Caskroom outside Homebrew. In the future, though (say a year or two, when most people have migrated to the new system and don’t have lingering app symlinks), I wonder if it won’t make more sense to go back inside Homebrew.
Either way, I’m ready to close this when anyone else is. It seems to be mostly resolved and pretty quickly contained (thank you for that).

Having everything inside of Homebrew (the directory) makes /usr/local just a bit cleaner, does not require sudo, and it makes some sense to have Cellar and Caskroom together. On the other hand, both projects do behave differently and install different types of apps in different ways, so having Caskroom separate from Homebrew also makes some sense.

Both the /usr/local/Homebrew and the /usr/local/Caskroom migrations require you to have ownership of /usr/local at the time of the migration.

In the future, though (say a year or two, when most people have migrated to the new system and don’t have lingering app symlinks), I wonder if it won’t make more sense to go back inside Homebrew.

I don't think it would. If it was to go to anywhere I'd suggest somewhere inside /usr/local/var/homebrew. /usr/local/Homebrew is purely a repository and we've worked pretty hard to avoid storing any data or state in there.

On the other hand, a user typing brew cask install google-chrome the first time and being prompted for the sudo password seems a bit weird (the former was explicitly a setup process, unlike the latter, though this all goes out the window when installing pkgs).

I agree here although longer-term I'd rather see Homebrew Cask where possible ask the user to run the sudo command rather than doing it for them. That removes the possibility of environment or parameter injection allowing another script to use Homebrew Cask as a privilege escalation vector.

I agree here although longer-term I'd rather see Homebrew Cask where possible ask the user to run the sudo command rather than doing it for them. That removes the possibility of environment or parameter injection allowing another script to use Homebrew Cask as a privilege escalation vector.

I'm more wary of this. The Homebrew install runs sudo for you. I'd rather have Cask use sudo for some small, discrete operation, than run brew cask as sudo and accidentally make my config files owned by root or something.

than run brew cask as sudo and accidentally make my config files owned by root or something.

That's not what I'm suggesting. I'm saying rather than cask running sudo command arg it instead aborts and tells the user to run sudo command arg.

I think if we do that, we’ll never hear the end of people asking for the old behaviour back. Though I understand your concern, we’d constantly be aborting users’ comands. Not forgetting that the commands to run can be a bit obtuse, and it’s a place for users to copy and paste wrongly and having nefarious resulta. It would also make diagnosing harder. Just thinking about it, the frustration is palpable.
We had plans (thinking mostly of people who do not have sudo access) for a flag that would abort commands that required sudo. In the end, that was scrapped for lack of interest.
We can certainly think of reinplementing that, but making it the default would just serve to annoy users. Just like when we had ~/Applications as the default directory most people used /Applications.

We haven’t had reports of this in a while.


Remote ADD should use HTTP cache headers

The current ADD implementation for remote files seems (see comment https://github.com/dotcloud/docker/blob/4f53722dee6c8ea18a9c30bde2cc6fee24b03d32/buildfile.go#L376 ) to download the actual file and then calculate its checksum to determine if it is a cache hit or miss.
It should trust the http headers instead to build the checksum to save time and bandwidth (if the server responds that the file has not changed, there is no need to download it again).
Currently (client version 0.7.6), using a command to download and process the file is a workaround:
RUN apt-get install wget
RUN wget -O - http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.0.tar.gz | tar xz

@duglin no. I'd rather have a slow builder than a broken one.

Well, a use case we have at the BBC is to download pre-computed data models to build data-only volumes. To be honest, when you get 16GB from Amazon S3, you feel a massive pain, especially because of a typo in the last filename.
Amazon S3 provides proper headers:
curl --head https://s3-eu-west-1.amazonaws.com/.../dbpedia_index.db
HTTP/1.1 200 OK
x-amz-id-2: FfrZ6IOlKapHyfs2hcx9IAnMhBAwXlQxpc77fQ5f4cZ0xt06NpIARMANO/EU6zakJ/g0sRnDxvo=
x-amz-request-id: 50132DAA1E90E2A7
Date: Wed, 21 Jan 2015 20:06:23 GMT
Last-Modified: Thu, 26 Jun 2014 09:17:04 GMT
ETag: "20fe762a4bf2340164a65f73acb517c1-91"
Accept-Ranges: bytes
Content-Type: application/octet-stream
Content-Length:<PHONE_NUMBER>
Server: AmazonS3

Is there a way to take in account the headers when they exist, and to drop any kind of cache (actual behaviour) when you judge it is unsafe to rely on them?

@erikh Using E-Tag if it exists with If-Modified-Since would mean you'd never get a 'bad build'; but you can skip hashing of the contents etc. Navigate to a page you've been to before in Firefox and you can see the Not Modified responses not being broken.
+1 to this when the server serves ETags.

Is there another Issue that tracks this feature request? All related I found seem to be closed but the issue still exists in 1.8.1

@dkirrane Reported again here: https://github.com/docker/docker/issues/15717


nrf9160 Non-Secure: use TF-M by default, and flash the combined binary when executing tests

Adding TF-M as the default Secure Firmware solution for nRF9160 Non-Secure builds.

Requires re-sizing the default secure partition size in device tree, to match the TF-M secure firmware size.

The current CI failures relate to the FP ABIs, so I am going to leave this to draft state until the FP issue is resoved.


Manage cookbook dependencies using berkshelf

Added Berksfile in all the cookbooks to correctly manage the cookbook dependencies.
This fixes issue #101

Currently berkshelf does not correctly handle transitive dependencies for cookbooks cloned from git. Hence, a cookbook, say 'G', downloaded from git needs to be specified in Berksfile of cookbook, say 'A' that needs the git cookbook and in other cookbooks, say 'B' and 'C' where the dependency relation is of the form C -> B -> A -> G (where '->' == 'depends on').

Should this PR also cleanup the code in setup_chef_cookbooks.sh which downloads these dependent cookbooks. If yes. we need to add a step to berks install and vendor the cookbooks before setting up the cluster.

The changes have been tested by building a vbox vm cluster and by my Jenkins server. Good for review and merge. I can squash commits further if desired.

The centos cookbook should be gone from #103 now

PR #616 supercedes this one.


Add module rational-persistence.el (reopen)

Sorry, i am new to all this, renamed my master branch because i initially din't branch for adding the module. Reopen of #75 ...   :^|
Some settings related to persistence of emacs over different sessions.
I like these a lot and didn't know about them for the longest time.
Maybe this is a way to expose these settings for users that are interested in this behaviour.


add support for TOML?

Would you mind me adding support for TOML? Because TOML looks simpler and less error-prone
There is a good article explaining the reason why TOML is better than YAML
https://arp242.net/weblog/yaml_probably_not_so_great_after_all.html

This is actually already doable via loaders.  This functionality was recently added by davidtheclark in this pull request.  If anything about it is unclear please let me know and I'll try to help out.

@olsonpm cool! I wonder if we can make TOML supported by default instead of requiring additional setting
because this library is required by so many useful tools (such as stylelint) and as far as I see they just use the default setting, it will be nice that TOML can also support by default!

Thanks @foray1010. I like the article: I find it convincing and I'll try using TOML more often :)
@olsonpm is right, though, that we won't be adding another default for TOML — not unless we see common reliance on it among cosmiconfig users. The "users" of cosmiconfig are not, however, the end users of tools like stylelint, but the authors of tools like stylelint. If stylelint users want TOML configuration, that's a feature request for stylelint; if lint-staged users want TOML configuration, that's a feature request for lint-staged. If stylelint and lint-staged and some other users then express that they'd like cosmiconfig to turn on TOML by default, we'd consider it.

because this library is required by so many useful tools (such as stylelint) and as far as I see they just use the default setting, it will be nice that TOML can also support by default!

Cosmiconfig should not be in control of this — that is, should not be making the decisions about adding more default configuration formats to libraries, unless those libraries want them.
So I'm happy to leave this issue open to see if library authors do want to add TOML to the defaults.

Chiming in here just to say that I also think toml is superior to json and yaml. The Rust (cargo.toml) and Python (pyproject.toml) community seems to think so too.

I'd love default toml support for Zoe.

More arguments in favor of TOML: lucidchart.com/techblog/2018/07/16/why-json-isnt-a-good-configuration-language

Maybe you'd be interested in confinode, as adding TOML support is straightforward with it (see here).

Just checking if there is any new opinions on this, I also need TOML and I'd reckon at this point it's caught some more traction enough to justify it as part of default loaders.

Sounds like a good idea. What's the go-to lightweight parser for this?
The apparently most popular parser, @iarna/toml, which prettier also seems to use, does not support the latest version of the spec.

Hi there, I see that this issue seem to be currently blocked by the lack of libraries. I don't want it to sound like self-promo but I'm the maintainer of https://www.npmjs.com/package/smol-toml, a lightweight TOML parser compliant with the 1.0.0 spec. Some prominent tools such as knip, rust-cache and unjs's confbox have adopted it, so I think it has seen quite a handful of real-world TOML files and I haven't heard about parsing issues for a while (which is something I'm happy about!).
I think it can fit the needs of cosmiconfig and bring native TOML support to this already very handy lib


Incomplete instructions - report suspicious activity

On the Authentication Methods settings page there is a Reporting Code option which is not described in these instructions. Please add some information about how that code is supposed to be implemented.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 84b26750-880c-8f14-1628-436293de915e
Version Independent ID: 518b727f-79e8-b5c2-bcb3-ff4de3444c6f
Content: Configure Azure AD Multi-Factor Authentication - Microsoft Entra
Content Source: articles/active-directory/authentication/howto-mfa-mfasettings.md
Service: active-directory
Sub-service: authentication
GitHub Login: @Justinha
Microsoft Alias: justinha

@DeanGross
Thanks for your feedback! We will investigate and update as appropriate.

sorry for delay and thanks for raising this. I created an internal work item for SMEs to review and update the topic as needed. #please-close

Hello, this topic is still open and I got the question from the customer what this "reporting code" is about / used for


script: Use flynn fork of godeb

godeb hasn't been updated to reflect the shutdown of code.google.com
for now use the Flynn fork at https://github.com/flynn/godeb

LGTM


feat: support the  element

Adds processing for the <picture> element
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/picture
resolves #3

Thank you @killtheliterate


tools: Fix fpgainfo security command to support both latest and older…

… sysfs directory structure
In linux-dfl, the security subdirectory moved out of the class driver and into the
device-specific parent driver. The code is modified to:

use sysfs recursive search
use *flash_count glob to satisfy both user_flash_count and flash_count file names

Signed-off-by: dbiswas-dev<EMAIL_ADDRESS>

Coverage increased (+0.03%) to 66.249% when pulling a57118ddbb09bcc728393817ba001baa69341445 on dbiswasx/fpgainfo_security_fix into 9c04849cf7caeeb88b480bd057bce00c44c184c5 on master.


A question about template usage, need guidance

https://github.com/EOSIO/eos/blob/24b98627ff4f6c0aaba74795ae1321fef68ff9be/contracts/dice/dice.cpp#L351
     bool has_offer( const checksum256& commitment )const {          auto idx = offers.template get_index<N(commitment)>();          auto itr = idx.find( offer::get_commitment(commitment) );          return itr != idx.end();       }
I have never saw such usage of template.
auto idx = offers.template get_index<N(commitment)>();
is 'template' the member of 'offers'?
or as a type?
or as a class/structure?
or as a keywords?
sorry for such basic question, thanks!

https://stackoverflow.com/questions/8463368/template-dot-template-construction-usage
my friends forward this link to me just minutes after I post this question, this issue should closed.


"hub issue show" throws error: unknown subcommand show

Problem
The man pages for hub shows
NAME
       hub-issue - Manage GitHub Issues for the current repository.

SYNOPSIS
       ...
       hub issue show [-f FORMAT] NUMBER
       ...

COMMANDS
       ...
       show
           Show an existing issue specified by NUMBER.
       ...

However:
$ hub issue show 1
error: Unknown subcommand: show

Expected
Expected hub to show content of the issue specified.
Actual
Error message: error: Unknown subcommand: show
Debug Info
$ hub version
git version 2.21.0
hub version 2.5.1

Additional Info

hub issue show was added to hub 2.6.0 more than 6 months ago. You have an outdated version of hub, but somehow you have man pages for a newer version of hub. You might have duplicate installations on your system. Check with which -a hub.
Solution: remove duplicate installations from your system and upgrade to the latest version of hub.


[Feature][FileSource&FileSink] Read or write files with specified character encoding

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
Read or write files with specified character encoding
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Does it mean to specify encodings such as UTF-8, ISO-8859-1, etc. when reading and writing files?

Does it mean to specify encodings such as UTF-8, ISO-8859-1, etc. when reading and writing files?

Yes.

Can you assign it to me please? I will try to implement it😄


Angular logout method (fixes #599)

Description:
Angular logout method inside AuthService
New imports / dependencies:

N/A

Unit Tests:

N/A

What tests do I need to run to validate this change:
Go to Angular-based view (right now only /o-nas) and logout already logged in user. It will not redirect You, it will just change header and all other user-dependent things.

Codecov Report

Merging #708 into master will decrease coverage by 0.13%.
The diff coverage is 20%.

@@            Coverage Diff             @@
##           master     #708      +/-   ##
==========================================
- Coverage   91.77%   91.63%   -0.14%
==========================================
  Files          88       88
  Lines        2661     2666       +5
  Branches        4        5       +1
==========================================
+ Hits         2442     2443       +1
- Misses        213      217       +4
  Partials        6        6

Impacted Files
Coverage Δ

frontend/src/app/auth.service.ts
30.76% <20%> (-2.57%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4508d75...1fc601d. Read the comment docs.


Issue with snapshot using delta on spark-dbt

Describe the bug
Hi All,
Im creating snapshot using Delta using local spark. But after the Initial snapshot, from the second dbt snapshot command, im getting the error snapshot target is not a snapshot table (missing "dbt_scd_id", "dbt_valid_from", "dbt_valid_to").
Im guessing this is because the select statement doesnt contain these fields as these are generated by dbt. But is there something Im missing or do we need to add these to select statement manually  from second run ? Any help would be great !
Steps To Reproduce

Connect dbt to Spark Thrift server
create a snapshot query using delta with followiing configs :
file_format=delta
unique_key= your unique key
strategy= timestamp
updated_at = your timestamp column

Expected behavior
scd2 operation on all dbt snapshot  queries
Screenshots and log output
target is not a snapshot table (missing "dbt_scd_id", "dbt_valid_from", "dbt_valid_to") from second dbt snapshot command
System information
Which database are you using dbt with?

[ ] postgres
[ ] redshift
[ ] bigquery
[ ] snowflake
[x] other (specify:  spark ( local ))

The output of dbt --version:
0.20.0

The operating system you're using:
RHEL 7 , Ubuntu etc
The output of python --version:
python3.9.6, python3.8
Additional context
Add any other context about the problem here.

@ashishmgofficial Thanks for reopening over here! I think the dbt-spark angle is the right way into this one.
Could you share the specific SQL from the initial snapshot (first dbt snapshot)? You can find it in logs/dbt.log.
I'd expect you to see a create table as statement, and for that statement to include the columns dbt_scd_id, dbt_updated_at, dbt_valid_from, and dbt_valid_to. Here's a silly example of my own:
-- snapshots/my_fun_snapshot.sql
{% snapshot my_fun_snapshot %}

{{
    config(
      target_schema='snapshots',
      unique_key='id',
      file_format='delta',
      strategy='timestamp',
      updated_at='updated_at',
    )
}}

select 1 as id, current_timestamp as updated_at

{% endsnapshot %}

-- copied from logs/dbt.log
-- trimmed whitespace for clarity
create or replace table snapshots.my_fun_snapshot
    using delta
    as
    select *,
        md5(coalesce(cast(id as string ), '')
         || '|' || coalesce(cast(updated_at as string ), '')
        ) as dbt_scd_id,
        updated_at as dbt_updated_at,
        updated_at as dbt_valid_from,
        nullif(updated_at, updated_at) as dbt_valid_to
    from (
        select 1 as id, current_timestamp as updated_at
    ) sbq

It sounds like your subsequent dbt snapshot is finding those columns missing from the snapshot table, so that initial run and create table statement are going to be key.

Hi @jtcohen6 , Following is the create table command form logs :
create or replace table snapshots_new_1.my_fun_snapshot

    using delta

    as

    select *,
        md5(coalesce(cast(id as string ), '')
         || '|' || coalesce(cast(updated_at as string ), '')
        ) as dbt_scd_id,
        updated_at as dbt_updated_at,
        updated_at as dbt_valid_from,
        nullif(updated_at, updated_at) as dbt_valid_to
    from (

    SELECT
        1 AS id,
        CURRENT_TIMESTAMP AS updated_at
    ) sbq

Also , Im testing this both in my local WSL with python3.8 as well as in a organization provided RHEL7 system with python3.9
in WSL :
Following error also shows up in the log recently :
/* {"app": "dbt", "dbt_version": "0.20.0", "profile_name": "default", "target_name": "dev", "connection_name": "list_None_curated_source_dbt_test__audit"} */
show table extended in curated_source_dbt_test__audit like '*'

2021-08-03 06:48:23.076050 (ThreadPoolExecutor-1_0): Database Error
  org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'curated_source_dbt_test__audit' not found
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:361)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263)
  	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78)
  	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258)
  	at java.base/java.security.AccessController.doPrivileged(Native Method)
  	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
  	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272)
  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:829)
  Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'curated_source_dbt_test__audit' not found
  	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireDbExists(SessionCatalog.scala:192)
  	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:951)
  	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:932)
  	at org.apache.spark.sql.execution.command.ShowTablesCommand.$anonfun$run$42(tables.scala:868)
  	at scala.Option.map(Option.scala:230)
  	at org.apache.spark.sql.execution.command.ShowTablesCommand.run(tables.scala:868)
  	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
  	at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
  	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
  	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
  	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
  	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
  	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
  	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
  	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
  	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
  	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
  	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
  	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:325)
  	... 16 more

2021-08-03 06:48:23.076894 (ThreadPoolExecutor-1_0): Error while running:
macro list_relations_without_caching
2021-08-03 06:48:23.077663 (ThreadPoolExecutor-1_0): Runtime Error
  Database Error
    org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'curated_source_dbt_test__audit' not found
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:361)
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263)
    	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78)
    	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62)
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263)
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258)
    	at java.base/java.security.AccessController.doPrivileged(Native Method)
    	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272)
    	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'curated_source_dbt_test__audit' not found
    	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireDbExists(SessionCatalog.scala:192)
    	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:951)
    	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:932)
    	at org.apache.spark.sql.execution.command.ShowTablesCommand.$anonfun$run$42(tables.scala:868)
    	at scala.Option.map(Option.scala:230)
    	at org.apache.spark.sql.execution.command.ShowTablesCommand.run(tables.scala:868)
    	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    	at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
    	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
    	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
    	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
    	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
    	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
    	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
    	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
    	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
    	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
    	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
    	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
    	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
    	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
    	at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
    	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:325)
    	... 16 more

curated_source is the default schema i had provided in the profiles.yml in .dbt folder. But i have provided custom schema for the snapshot also . Following is my snapshot sql :
{% snapshot my_fun_snapshot %}
    {{ config(
        target_schema = 'snapshots_new_1',
        unique_key = 'id',
        file_format = 'delta',
        strategy = 'timestamp',
        updated_at = 'updated_at',
    ) }}

    SELECT
        1 AS id,
        CURRENT_TIMESTAMP AS updated_at
{% endsnapshot %}

This i have done on a fresh dbt project with the snapshot sql alone in the snapshots folder
In RHEL7:
Im not getting any errors like the above in the logs .. just the final Compilation error message
Im using a docker based setup to start up following services :

minio
mariadb ( external metastore )
Hive metastore standalone

my spark-defaults.conf :
spark.sql.extensions             io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog  org.apache.spark.sql.delta.catalog.DeltaCatalog
hive.metastore.uris              thrift://<IP_ADDRESS>:9083/
spark.sql.warehouse.dir          s3a://spark/warehouse/
spark.hadoop.fs.s3a.endpoint     http://<IP_ADDRESS>:9000
spark.hadoop.fs.s3a.impl         org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.access.key   accesskey
spark.hadoop.fs.s3a.secret.key   secretkey
spark.driver.extraClassPath      /opt/spark/lib/*
spark.executor.extraClassPath    /opt/spark/lib/*
spark.executor.cores             3
spark.driver.cores               2

Using this I start my $SPARK_HOME/sbin/start-thriftserver.sh

I came across this issue sometime in June when I tried to use dbt-sparkv0.20.0 which has thrift https implementation. Snapshot  works fine without any issue in v0.19
v0.19 get_columns_in_relation  method directly calls the macro spark__get_columns_in_relation which executes the statement describe extended tablename. This sql stmnt would return the table schema with column names and data type for a delta table
From 0.20 get_columns_in_relation method has new logic to check the cached schema 1st to get the column list if relation is present in cache. This relation cache is built during the initial process using macro spark__list_relations_without_caching
This macro executes show table extended in databasename like '*' spark sql query to get all the tables and its schema under the databasename. Unfortunately this spark sql output doesnt provide the schema details of delta table (not sure if its due to open source version of delta).
Check below examples where I created two tables, one delta format and other one parquet format.
The show table extended output provide the schema for parquet table but not for delta table.
>>> spark.sql("""
... CREATE TABLE events_delta (
...   date DATE,
...   eventId STRING,
...   eventType STRING,
...   data STRING)
... USING DELTA
... """)
DataFrame[]

>>> spark.sql("""
... CREATE TABLE events_parquet (
...   date DATE,
...   eventId STRING,
...   eventType STRING,
...   data STRING)
... USING PARQUET
... """)
DataFrame[]

>>> spark.sql("SHOW TABLE EXTENDED  like '*'").show(truncate=False)
+--------+--------------+-----------+------------+
|database|tableName     |isTemporary|information |
+--------+--------------+-----------+------------+
|default |events_delta  |false      |Database: default
Table: events
Owner: vscode
Created Time: Wed Aug 04 04:29:25 UTC 2021
Last Access: UNKNOWN
Created By: Spark 3.1.1
Type: MANAGED
Provider: delta
Location: file:/workspace/spark-warehouse/spark-warehouse/events
Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Partition Provider: Catalog
|

|default |events_parquet|false      |Database: default
Table: events_parquet
Owner: vscode
Created Time: Wed Aug 04 04:32:25 UTC 2021
Last Access: UNKNOWN
Created By: Spark 3.1.1
Type: MANAGED
Provider: PARQUET
Location: file:/workspace/spark-warehouse/spark-warehouse/events_parquet
Serde Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
Schema: root
 |-- date: date (nullable = true)
 |-- eventId: string (nullable = true)
 |-- eventType: string (nullable = true)
 |-- data: string (nullable = true)

|

But describe extended provides the schema details for delta tabe.
>>> spark.sql("describe extended events_delta").show(truncate=False)
+----------------------------+----------------------------------------------------------------+-------+
|col_name                    |data_type                                                       |comment|
+----------------------------+----------------------------------------------------------------+-------+
|date                        |date                                                            |       |
|eventId                     |string                                                          |       |
|eventType                   |string                                                          |       |
|data                        |string                                                          |       |
|                            |                                                                |       |
|# Partitioning              |                                                                |       |
|Not partitioned             |                                                                |       |
|                            |                                                                |       |
|# Detailed Table Information|                                                                |       |
|Name                        |default.events                                                  |       |
|Location                    |file:/workspace/spark-warehouse/spark-warehouse/events          |       |
|Provider                    |delta                                                           |       |
|Table Properties            |[Type=MANAGED,delta.minReaderVersion=1,delta.minWriterVersion=2]|       |
+----------------------------+----------------------------------------------------------------+-------+

As the relation exists in cache, get_columns_in_relation method returns the schema from cache which doesnt have any column from the target snapshot table.
valid_snapshot_target method checks if standard snapshot columns "dbt_scd_id", "dbt_valid_from", "dbt_valid_to" are present in the target or not and raises an exception saying target is not a valid snapshot table.
Due to this new change in v0.20.0, very 1st run of snapshot where target table doesnt exists will work and next snapshot run would fail saying target is not a valid snapshot table

@harryharanb Ah, thanks for the detailed report! I had a suspicion this may be related to the changes to get_columns_in_relation in v0.20, and you make a compelling argument.
If that's the case, I'm hopeful that #197 offered a fix for it. You can try it out by installing the patch release candidate (with a final release coming soon):
pip install dbt-core==0.20.1-rc1
pip install dbt-spark[PyHive]==0.20.1-rc1

@jtcohen6 - Thanks for the details.  I saw that fix but not sure if its addressed for this delta column missing issue. That issue is when the table is created in same run and information is missing. But here if you see the query output, information column is populated. but its just that its missing schema details.
so not sure the fix cached_relation.information is None will satisfy and goes into super().get_columns_in_relation(relation) or not.
I will still try the fix and update here.

@jtcohen6  and @harryharanb I tested the same using 0.20.rc1 today. As expected, its not working . As pointed out by @harryharanb , its working fine using 0.19.2 though

@harryharanb @ashishmgofficial Thanks for testing this out, and for thinking through this together.

Unfortunately this spark sql output doesnt provide the schema details of delta table (not sure if its due to open source version of delta).

You might be onto something. Personally, I've been able to run snapshots using dbt-spark==0.20.0 just fine. When I run show table extended like '*' against the Databricks test environment we use, it does include the column schema:
Database: snapshots
Table: my_fun_snapshot
Owner: root
Created Time: Tue Aug 03 01:58:21 UTC 2021
Last Access: UNKNOWN
Created By: Spark 3.0.1
Type: MANAGED
Provider: delta
Location: dbfs:/user/hive/warehouse/snapshots.db/my_fun_snapshot
Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Partition Provider: Catalog
Schema: root
 |-- id: integer (nullable = true)
 |-- updated_at: timestamp (nullable = true)
 |-- dbt_scd_id: string (nullable = true)
 |-- dbt_updated_at: timestamp (nullable = true)
 |-- dbt_valid_from: timestamp (nullable = true)
 |-- dbt_valid_to: timestamp (nullable = true)

Could we fix this by adding another check, to make sure that cached_relation.information not only exists but contains information about the column schema? I'm thinking we could change this line:
https://github.com/ali-tny/dbt-spark/blob/4f12570219f8159a5b1e5751418a90c7662c6f42/dbt/adapters/spark/impl.py#L215
To be:
        if (
            cached_relation is None or
            cached_relation.information is None or
            len(self.parse_columns_from_information(cached_relation)) > 0
        ):

@jtcohen6 - that would work. but on databricks environment self.parse_columns_from_information would run twice.
May be something like below.
columns = []
if cached_relation and cached_relation.information:
  columns = self.parse_columns_from_information(cached_relation)
if not columns:
    rows: List[agate.Row] = super().get_columns_in_relation(relation)
    columns = self.parse_describe_extended(relation, rows)

Also looks like this an open issue issue#662 with open source deltalake. I used delta-core_2.12-1.0.0.jar for my testing and issue still persist.

Right on, no sense running it twice if we don't have to. Thanks for finding + linking that Delta issue!
@harryharanb @ashishmgofficial Would either of you be open to contributing this fix? :) I'd love to get it in for v0.20.1 (final).

@jtcohen6  - I can pick this up, but I would be able to test only with open source deltalake. Also this would be my 1st time contributing to open source community. Do you have any document/instructions for PR process?

@harryharanb Hooray! Your ability to test with open source deltalake is much appreciated, since that's where it seems this error is cropping up. (We unfortunately don't have open source deltalake set up in our CI pipeline today.)
Could you:

fork this repo
make the code change you suggested above
commit + push to a branch on your fork
open a PR from the fork branch to the master branch in this repo

So long as the existing tests all pass, we can work on adding a changelog entry and getting the PR merged.

@jtcohen6 thanks for the details. I will work on this over the weekend

@harryharanb Is there any update on this ?

@ashishmgofficial raised PR and all tests are passed. Lemme know if its good to merge this for 0.21 release.


Domain Example

Hey 👋
Love the work!
Do you have an example of multi tenancy via domains with individual databases? For example:
Main domain: mysaasapp.com
Tenant domain: tenantname.mysaasapp.com
Tenant 2 domain tenanttwoname.mysaasapp.com
etc? From what I read, its supported, but unsure on implementation.

change this line
https://github.com/go-saas/saas/blob/0b157bcf9a4e237767611149f972a4a9ff4628d9/examples/gorm/main.go#L136-L137
to
import(
   ...
   shttp "github.com/go-saas/saas/http"
   ...
)
...
r.Use(sgin.MultiTenancy(tenantStore,
		sgin.WithMultiTenancyOption(shttp.NewWebMultiTenancyOption("", "([-a-z0-9]+)\\.mysaasapp\\.com"))))

adjust your own tenant name regex format
see https://github.com/go-saas/saas/blob/main/http/domain_tenant_resolve_contrib.go


Downloaded maps do not show up in offline maps.

Describe the bug
Mapbox downloads are successful but MapWarper downloads fail. Mapbox downloads, though complete the download, are not available in offline maps.
Are you able to reproduce the problem in a new project after restarting the app?
Yes (As per the comments in dropbox)
Screenshots

Fixed.


Internationalized Hardcoded Strings

Contains
Internationalize some hardcoded strings in the UI
How to test

Translate to another language.
Check whether or not it's translated inside the game.

Can one of the admins please verify this patch?


UAT | Hearing | When judge joins the hearing he/she should able to get proper case name.

Describe the bug
When judge joins the hearing he/she should able to get proper case name
To Reproduce
Steps to reproduce the behaviour:
1.Login as judge
2.Click on Click on 'View Hearings' button
3.Select any hearing date
4.Click on 'Join hearing' button
Expected behaviour
When judge join , the hearing he/she should able to get proper case name.
Actual behaviour
For each case in the hearing getting same name as 'Aparna vs. Subarna'
Desktop (please complete the following information):
OS: windows
Browser : chrome

Tested in QA env, Working as Expected. Case name is showing inside the Hearing


support processing other packet from ipstack

It is better to create two types for the IpStackStream: one for UnknownTransport, which we know is IP but is neither TCP nor UDP, and another for UnknowNetwork for anything that is not IP. I believe these types allow users to implement their protocols manually.
Since ICMP is not primarily designed to transfer data, we have to create an enum that supports the most useful types of ICMP in the IpStack usage scenarios, such as Echo.

Something like this?
https://github.com/narrowlink/ipstack/commit/670226a9565655c8dcbc927582012e48b2c56580

I think providing the raw packet produced bu tun for all  the other cases other than tcp and udp has the maximum flexible because we may want to do the following things:

process the packet by using the familiar crate.
send the complete packet to the remote to be processed.

@xmh0511: could you please check examples/tun.rs in the main branch?
https://github.com/narrowlink/ipstack/blob/main/examples/tun.rs#L113C1-L134

@SajjadPourali  I have tried the latest version. Currently, in the branch IpStackStream::UnknownTransport(u), u.payload() will return the payload part of the IP packet, which is the ICMP message, and u.send just need the ICMP message too, right?

@SajjadPourali I have tried the latest version. Currently, in the branch IpStackStream::UnknownTransport(u), u.payload() will return the payload part of the IP packet, which is the ICMP message, and u.send just need the ICMP message too, right?

True, there also another one that called UnknownNetwork that is similar to the raw that you mentioned.


ing cached gradio-4.31.5-py3-none-any.whl.metadata (15 kB) ERROR: Could not find a version that satisfies the requirement bitsandbytes==0.43.1 (from versions: 0.31.8, 0.32.0, 0.32.1, 0.32.2, 0.32.3, 0.33.0, 0.33.1, 0.34.0, 0.35.0, 0.35.1, 0.35.2, 0.35.3, 0.35.4, 0.36.0, 0.36.0.post1, 0.36.0.post2, 0.37.0, 0.37.1, 0.37.2, 0.38.0, 0.38.0.post1, 0.38.0.post2, 0.38.1, 0.39.0, 0.39.1, 0.40.0, 0.40.0.post1, 0.40.0.post2, 0.40.0.post3, 0.40.0.post4, 0.40.1, 0.40.1.post1, 0.40.2, 0.41.0, 0.41.1, 0.41.2, 0.41.2.post1, 0.41.2.post2, 0.41.3, 0.41.3.post1, 0.41.3.post2, 0.42.0) ERROR: No matching distribution found for bitsandbytes==0.43.1

I get this when I try to install it

I got same. Are you on mac? It would be helpful to provide information about your python version, operating system, etc. Otherwise it is impossible to debug your situation.

x86_64 GNU/Linux
ERROR: bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.


missing "initialize super" call in unreferenced class causes error message for some other class

The Crystal compiler is reporting a "not initialized in all 'initialize' methods" error in a class that appears to be initializing things correctly. It turns out that the cause of the problem was in a separate, not-directly-referenced class (that is, the class was not mentioned again in the program but was referenced indirectly in a Ruby-esque sense). The error made no mention of this other class, so it took some tracking to figure out.
I'm hoping the error message for this case could be improved.
(I couldn't find a similar report here or on the forum, but sorry if this has already been discussed.)
Here is the simplest form of the problem I could come up with.

class Base
  property x

  def initialize()
	@x = 0
  end
end

#No (direct) references to this class in the code
class Unreferenced < Base
  def initialize()
	#forgot the super call
  end
end

class Derived < Base
  def initialize()
	super()
  end

  def print_x()
	print @x + 100
  end
end

b = Derived.new
b.print_x

Crystal 0.7.6 produces the following error:
Error in line 28: instantiating 'Derived#print_x()'

in line 23: undefined method '+' for Nil (compile-time type is (Nil | Int32))

================================================================================

Error: instance variable '@x' of Derived was not initialized in all of the 'initialize' methods, rendering it nilable.

Specifically in this one:

  eval:18

I have found three ways to fix the error:

Remove the Unreferenced class.
Change Unreferenced < Base to just Unreferenced.
Add a super call to Unreferenced's initialize.

I get the general idea that the definition of Unreferenced affects the definition of Base. However, the error message only mentions my Derived class, and mentions of Nil and nilable reinforce (in mind mind, at least) the notion that something is wrong in Derived rather than in Base or Unreferenced. These factors made an otherwise simple problem difficult to track down.
I'm certainly more conscious of this type of error now, but I'm wondering if there is room for improvement somewhere.

Is it possible to provide a more specific error message for this case? Or maybe provide more information when running with --verbose? I would not have thought to look down to Base and then back up to Unreferenced, so something along these lines would have been helpful.

Is it possible (and advisable) to change the way I define classes to prevent this type of error from occurring?

Thanks.

The error message is wrong, it should point to the initialize without super. It could also suggest "did you forget to call super?", maybe.

Great, thanks. I was afraid it was one of those things that makes sense to everyone but me. :smiley:

Now for your example it says:
Error in ./foo.cr:27: instantiating 'Derived#print_x()'

b.print_x
  ^~~~~~~

in ./foo.cr:22: undefined method '+' for Nil (compile-time type is (Nil | Int32))

    print @x + 100
             ^

================================================================================

Error: instance variable '@x' of Unreferenced was not initialized in all of the 'initialize' methods, rendering '@x' of Derived nilable (Base is the common supertype that defines it)

Specifically in this one:

  ./foo.cr:11

      def initialize()
          ^~~~~~~~~~

That looks fantastic, thanks a lot.


upgrade to Apache Bookkeeper 4.7.3 to fix DistributedLog based stateful storage

This fixes #3210

upgrade to apache/bookkeeper:4.7.3 to allow DlogStorage successfully persist stateful state to bookie
update variety of components dependent on DistributedLog API to org.apache.distributedlog:distributedlog-core-shaded:4.7.3
update those Kubernetes deployments that use Bookkeeper cluster

@nwangtw

LGTM.
We don't use dlog internally.  @jerrypeng  do you have time to double check? Thanks!

Thanks @jerrypeng ~

has there been a new heron helm package released since these changes have been made? I would like to use the heron helm charts with apache bookkeeper 4.7.3, but I cant find any updated docs/packages. Just wanted to check before I resorted trying to package it myself. Thanks!

Not yet.
Dmitry is working on stmgr memory management and hopefully it fixes a mem leak. I think we can have another RC after the fix is merged: https://github.com/apache/incubator-heron/pull/3282

cool thanks, should i expect that to be put here? https://storage.googleapis.com/heron-charts

Interesting. I have no idea about that chart. :D  @kramasamy @jerrypeng  @nlu90 @huijunw any idea?

its referred to in this doc: https://apache.github.io/incubator-heron/docs/operators/deployment/schedulers/kubernetes-helm/

That was Streamlio’s chart repository.  I think we will have to do vote for an official release.  Otherwise you will have to build from source and create your own distribution.  We are in progress of updating the docs now.  There is an open PR for the website that has about half of the work done with a checklist of things to do for updating our docs.  Feel free to check it out.  The PR is here -> https://github.com/apache/incubator-heron/pull/3280

If I did want to build the chart myself, can I just replace VERSION with 0.17.8, remove ".template" from the Chart.yaml and values.yaml, put it in a folder called heron and run helm package? or are there more steps? asking because I did that and I'm getting "release heron failed: the server could not find the requested resource", wondering if I'm missing something.
Also as a side note, I have rook-ceph set up and was previously using that for my bookkeeper storage. Is there a way in the heron helm chart to change the storage class for the journal-disk and ledger-disk volumeClaimTemplates from default using the values.yaml?
Thanks again

I am wondering who is the owner of that file and if there is a log of changes in that file?

I did adapt the relevant Helm templates in this PR. So, theoretically, you should be able to install the chart from locally. But, the tricky part is that there's no post-0.17.8 official release available in public, which means no heron post-0.17.8 docker image in public.
So, what we did in our deployment is to build heron RC release, push the docker image built to our own Artifactory repository, also push the generated maven artifacts there. Then, in our k8s cluster (we're on-promise), we configure the cluster to include our own Artifactory docker image repo.
Then we maintain a slightly different version of Helm chart based on the bare metal flavor of the templates available in this code base, because we're on-promise, and we don't need to care about GKE, AWS. So, our in house helm chart is simplified, but fairly close to the one in the source.

sorry if this isnt the right place for this discussion at this point. I was working through the issue, I've hit a couple issues looking at the code on master that maybe have been fixed in another branch I dont know about.

Baremetal refers to ledger disk, everything else refers to data-disk
trying to run helm chart with baremetal fails because it doesnt know what data-disk is
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/bookie.yaml#L141

So I changed it to minikube and got past that error as a temp fix

my "the server could not find the requested resource" problem
I've narrowed my problem down to the line below
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/tools.yaml#L18

my theory, is that the"-" in "{{- $platform" and the one on the other side of ".Values.apiServerMemory -}}" removes all white space between the comments and "apiVersion: v1". The problem, I think, is that this removes so much whitespace that "apiVersion: v1" actually gets commented out, like so:
"#  under the License.apiVersion: v1"
removing the first "-" from "{{- $platform" caused my error to go away, as well as copying the "apiVersion: v1" line and pasting it directly below itself.
That at least gets me to the point where I can helm install and get new errors

Added storageClassName: nameOfMyStorageClass directly after each of these lines with the same indentation to use rook-ceph for my storage, ended up needing it for bookie and zookeeper. still using "minikube" as my platform so I get the volumeClaimTemplate stuff.
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/zookeeper.yaml#L148
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/bookie.yaml#L164
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/bookie.yaml#L175

This line was also an issue for me:
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/tools.yaml#L73
This seemed to work better for me:
command: ['sh', '-c', '/opt/bookkeeper/bin/dlog admin bind -l /ledgers -s {{ .Release.Name }}-zookeeper:{{ .Values.zookeeper.clientPort }} -c distributedlog://{{ .Release.Name }}-zookeeper:{{ .Values.zookeeper.clientPort }}/heron']

sorry again if im spamming this, I just want to keep track of these somewhere.
On a somewhat more personal preference note, I don't like using the latest tag so that I can better keep track of what versions of things I'm running. It would be a "nice to have" to let the kubectl image be configurable in the values yaml as well
https://github.com/apache/incubator-heron/blob/6ce013cf262b8d413aec8c6e628ce363ef717ff1/deploy/kubernetes/helm/templates/tools.yaml#L161

No worries. If you like, feel free to file an issue and we can continue the discussion.
agree that "latest" can be tricky.

netcat (nc) is either not installed or not in the path of the container apache/bookkeeper:4.7.3, so the init container steps below fail:
https://github.com/apache/incubator-heron/blob/9ffeb39aec98d96caf2311b240216c8df6a4cc65/deploy/kubernetes/helm/templates/bookie.yaml#L104-L120
tried using the section from the minikube bookkeeper, I get "Error not able to create the directory /bookkeeper/data/journal" and "Error formatting journal directory failed"
https://github.com/apache/incubator-heron/blob/master/deploy/kubernetes/minikube/bookkeeper.yaml#L54-L62
ended up removing that section, that seemed to work

This line needs to change from org.apache.heron.uploader.dlog.DLUploader to com.twitter.heron.uploader.dlog.DLUploader ( see https://github.com/apache/incubator-heron/issues/2913#issuecomment-448798235)
https://github.com/apache/incubator-heron/blob/master/deploy/kubernetes/helm/templates/tools.yaml#L136

so after those changes, everything comes up healthy with a helm install! I now get an error when i try to submit a example topology where it says "Path length must be > 0". any thoughts?

Which topology? Maybe be good to start with a basic one, like word count.
Also it could be helpful to see if the topology runs ok locally.
Another guess might be the config.


Green skill classifier

Description
Improves the way we map skills to ESCO green skills by training a classifier.

Adds a green skills classifier green_skill_classifier.py
Creates a new function .get_measures in SkillMeasures which inputs the job advert. Before this wasn't actually in a function, and there were 2 instances of repeating code to do it.
Changes to /skill_measures_utils.py to include the green skill classifier
Updates README
Adds a util function map_skills_utils.py to map the skills to all of ESCO (not just green ESCO). This is comprised of code copied from ojd_daps_skills (I've tried to make it clear what is copied). So doesn't really need reviewing. Its a temporary fix, but ideally in the future we'd change ojd_daps_skills to have this functionality - unless it turns out we never use the skills mapped to all of ESCO
skill measures test test_skill_measures.py
New get_embeddings function in bert_vectorizer.py - just because this code kept being repeated

Fixes #45 #27  #10 (since it uses ONET green topics in the classifier)
Instructions for Reviewer
In order to test the code in this PR you need to ...
See if the tests work:
pytest

Check out the new green skill classifier:
from dap_prinz_green_jobs.pipeline.green_measures.skills.green_skill_classifier import GreenSkillClassifier

green_skills_classifier = GreenSkillClassifier()
green_skills_classifier.load_esco_data()
green_skills_classifier.load(
    model_file="s3://prinz-green-jobs/outputs/models/green_skill_classifier/green_skill_classifier_20230906.joblib"
)

pred_green_skill = green_skills_classifier.predict(
    ["Excel skills", "Heat pump installation skills"]
)

>>> [('not_green', 1.0, ('carry out sample analysis', '82423b5c-486f-42e7-b00e-7358757a8de5', 0.2772792296638087)), ('green', 0.7191159533073931, ('heat pump installation', '00735755-adc6-4ea0-b034-b8caff339c9f', 0.9072619656040537))]

Check out the new green skills measures
from dap_prinz_green_jobs.pipeline.green_measures.skills.skill_measures_utils import (
    SkillMeasures,
)
sm = SkillMeasures(config_name="extract_green_skills_esco")
sm.initiate_extract_skills(local=False, verbose=True)

job_adverts = [
    {"id": 1, "job_text": "This job requires communication and Excel skills. We want someone with experience in sustainability. Heat pump installation skills would be useful. We have a pension scheme benefit."},
    {"id": 55, "job_text": "This job contains no skills."},
    {"id": 3, "job_text": "This job contains Excel skills."},
    {"id": 8, "job_text": "This job contains a really long sentence. Promote good practice of material sustainability (reuse and or recycle) initiatives to reduce waste and save costs."},
    ]

# We will load the green taxonomy embeddings from S3 since they have already been calculated
taxonomy_skills_embeddings_dict = sm.get_green_taxonomy_embeddings(
    output_path="outputs/data/green_skill_lists/green_esco_embeddings_20230815.json", load=True)

prop_green_skills = sm.get_measures(job_adverts)

Next:
After applying these changes to the mixed ojo sample, the new results for these green skills will be in outputs/data/ojo_application/extracted_green_measures/20230912/ojo_sample_skills_green_measures_production_False_base.json
I intend to rerun some of the analysis using these/quality check the most common green skills etc.
Checklist:

[ ] I have refactored my code out from notebooks/
[ ] I have checked the code runs
[ ] I have tested the code
[ ] I have run pre-commit and addressed any issues not automatically fixed
[ ] I have merged any new changes from dev
[ ] I have documented the code

[ ] Major functions have docstrings
[ ] Appropriate information has been added to READMEs

[ ] I have explained this PR above
[ ] I have requested a code review

thanks for the review @india-kerle ! I've addressed your comments and made the fixes @sqr00t suggested in Slack to the requirements.txt.

heya, just one smaller change needed: removing awscli from requirements_dev.txt


Question: what’s the best scope to address missing default export of Astro components?

I started using eslint-plugin-import  for my Astro pages, and when I try to import Astro components, the plugin reports: ESLint: No default export found in imported module "../../layouts/Layout.astro".(import/default)
My gut feeling tells me that astro-eslint-parser should report the default export as existing, but maybe it’s the import plugin that needs to be configured properly, or even eslint-plugin-astro.
In which repository should I file an issue for that?

eslint-plugin-import doesn't know about *.astro semantics, so I think we need to do something in astro-eslint-parser to resolve that.
Could you please provide a repository where we can reproduce the issue? I think we can use it as a test case.

Here you go https://github.com/ota-meshi/astro-eslint-parser/issues/259


[MVP] Migrate DP to Magma

LTE <> DP modules

Modules hierarchy: https://magma.github.io/magma/docs/next/orc8r/dev_dependencies
Plan is to utilize a separate DP module for the functionality that is specific to the DP domain
DP module will depend on LTE module, out of architectural necessity ( I.e. new module hierarchy will be DP -> LTE -> Orc8r)

Orc8r DP module

Straightforward, standalone K8s pods-backed services providing Orc8r targets for DP-enabled AGWs
Any number of services in the DP module is acceptable, especially for MVP, though preference for monolithic services
dp/gateway will contain no new services
dp/protos will contain the relevant protos for DP-domain data models and Orc8r<>AGW interface
dp/cloud will contain the DP-relevant Orc8r services

Current list of subtasks:

[x] Migrate DP code into proposed directory structure (in magma, this will be dp/cloud and dp/protos)
[x] Adjust the code to the new dir structure and ensure that Unit tests run locally
[x] Integrate Magma's DP with all essential services (at this point it seems its only psql database)
[x] Ensure that Magma's Postgresql as a DP database provisioned
[x] Add possibility of local development on EKS cluster
[x] Test orc8r deployment with dp (have DP services actually be deployed as part of orc8r)
[x] Unit tests for DP pass in CI
[x] Integration tests for DP pass in Magma environment (this will also require a spike to see how this could actually be done)

[x] Prepare job to deploy/update orc8r on AWS
[x] Prepare code to build and upload docker containers for orc8r
[x] Update helm chart to receive traffic from enodebd
[x] Prepare job to deploy/update dp on AWS
[x] Prepare job to upload helm chart to helm repository
[x] Prepare code to open reverse ssh tunnel to agw, during deployment job.
[x] Prepare code to deploy/update enodebd code on agw.
[x] Prepare code/job to run integration tests.
[x] Prepare code/job to collect integration tests results.

There is a complication regarding the migration.
It appears that we will not be able to deploy magmas "master" on FreedomFi lab (requires confirmation from FreedomFi).
Development workflow of:

Develop into Magmas Community "master" and merge
Cherry-pick from Magmas master into development branch of FreedomFi magma fork
appears to be very time consuming and not trivial.

This task will re re-estimated and planned after Sprint Review session.

https://github.com/openEPC/magma/pull/19

Adding a sub-task for the possibility of local development on EKS cluster (~2 SPs)

[Joey] For the DB objects creation let's try to use a user with the privileges limited to DP objects only to narrow possible damage if something goes wrong

Running integration tests on eks in Jenkins:
https://jenkins.freedomfi.com/job/domain_proxy/job/dp-integration-tests/job/main


Create stakesstonechain.json

I'm getting the following error trying to open a connection with your chain. Please make sure you have you gaia the version shown in https://github.com/iqlusioninc/relayer/tree/master/testnets
I[2020-04-27|21:13:32.413] ✘ [stakesstonechain]@{58100} - msg(0:update_client,1:connection_open_try) err(client: self consensus state not found)
{"height":"58100","txhash":"DFD908D17F546A559EACA366F0BD1D934A6962368B5282415AF3DF50E99310EE","codespace":"client","code":20,"raw_log":"failed to execute message; message index: 1: self consensus state not found","gas_wanted":"200000","gas_used":"112038"}

Hi Jack, I updated the version to the mentioned one in the github. Could you check it again? Thank you

Closing in favor of the https://github.com/cosmosdevs/GameOfZones repo.


Add files via upload

Added .md files for LoadFactor and DeltaProduction

Do not try to contribute multiple features via a single PR
Seperate LoadFactors and DeltaProduction in 2 PRs please
Files do NOT go into docs-kits_versioned_docskits\Demand and Capacity Management Kit...
Files need to go into docs-kits\kits\Demand and Capacity Management Kit...


AppView Proxy continues displaying "Load New Posts" popup.

Describe the bug
When the "AppView Proxy" toggle is enabled, the Load New Posts popup doesn't go away. Disabling the proxy solves the issue.
To Reproduce
Steps to reproduce the behavior:

Enable AppViewProxy

Expected behavior
To have it only show Load New Posts when new posts are available.
Screenshots

Details

Platform: Desktop Safari
Platform version: Version 17.0 (196<IP_ADDRESS>.4)
App version:Build version: 1.37.0; Platform: web

Additional context
EDIT: actually i think it might be showing that when there's updates to things like replies/likes/reposts

This has been fixed! #1065


(UI) onboard docs for JuiceFS

We want to onboard JuiceFS https://github.com/juicedata/juicefs

JuiceFS have custom URL mapping which is incompatible with the current docsQA impl.


Cannot get SimilatedDevice.py to run

This does not work on MacOS 10.14.3 with either Python 2.7, Anaconda 3.6 or Anaconda 3.7.
Lots of issues just getting the SimulatedDevice.py to work.
pip install azure-iothub-service-client
629  19/03/11 21:28:31 pip install azure-iothub-device-client
I tried using pip install of both azure-iothub-service-client, and azure-iothub-device-client - Library not loaded errors when I tried python SimulatedDevice.py.
I tried building the sdk from scratch - seg fault and
([100%] Linking CXX shared library iothub_service_client.dylib
ld: warning: -L path '/Users/xxxxxxx/anaconda3/lib/libpython3.7m.dylib' is not a directory)
Frankly - this was a non-starter and there were loads of posts on Stackoverflow with similar problems.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d9565ceb-e636-e815-f7cf-ff9523fabc2f
Version Independent ID: 7c6b3210-d878-0347-27af-199c2c00daa9
Content: Send telemetry to Azure IoT Hub quickstart (Python)
Content Source: articles/iot-hub/quickstart-send-telemetry-python.md
Service: iot-hub
GitHub Login: @wesmc7777
Microsoft Alias: wesmc

@jonmach We are aware of dependency limitations with the current builds of the library. These are noted here

Mac OS: Python 3.7.0 (or 2.7) + libboost-1.67 + curl 7.61.1 (all installed via homebrew) Any other distribution/OS probably embed different versions of boost & dependencies which won't work and will result in an ImportError at runtime.

We are fully aware that this is a less than ideal situation and we are currently working on a full python implementation that will have the wide platform support that is expected of any respectable python library. While we intend to support this wrapper-based version of the SDK until the new one is full-featured, we will be deprecating it shortly thereafter. We will update this readme with the link to the new preview version when it is available.

#please-close


python310Packages.caldav: 0.11.0 -> 1.0.1

This breaks home-assistant.tests.components.caldav.
Description of changes
https://github.com/niccokunzmann/python-recurring-ical-events/blob/v2.0.0/README.rst#changelog

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@ofborg build home-assistant-component-tests.caldav

@ofborg build home-assistant-component-tests.caldav

It worked with Home Assistant 2023.1.3.

@ofborg build home-assistant-component-tests.caldav

That's already broken on master.

Yeah, wondering whats the culprit.

Yeah, wondering whats the culprit.

I'm running git bisect but that might take a while.

I tried that, but it was annoying because so much stuff was broken and fixed along the way in python-updates.

Yeah, wondering whats the culprit.

Apparently it's d26402c2d9d623941cc6d294077de11d70ef21ea.

Result of nixpkgs-review pr 210299 run on x86_64-linux 1

7 packages built:

calendar-cli
gtg
home-assistant-component-tests.caldav
python310Packages.caldav
python310Packages.recurring-ical-events
python311Packages.caldav
python311Packages.recurring-ical-events

Cool, thanks!